Implement multipart/mixed parser

[Hendrik Cech]

Hey,

I'm currently trying to make multipart/mixed requests and parse the responses. The format is explained here: https://developers.google.com/storage/docs/json_api/v1/how-tos/batch

Implementing the body generator for the request wasn't too hard. Now I'm thinking about how to write a streaming response parser.

I would like to have an api similar to this:

http.request(opts, function(res) {

    var parser = new Parser()

    res.pipe(parser)

    parser.on('part', function(part) { … })

})

Each part of the response is a valid http response in itself. Is it therefore possible to use the core http module to parse the individual parts?

I'm not too eager to reimplement a http parser. Unfortunately I haven't found a module for that job either.

Thanks,

Hendrik

[mscdex]

On Sunday, April 6, 2014 9:26:24 PM UTC-4, Hendrik Cech wrote:

I'm currently trying to make multipart/mixed requests and parse the responses. The format is explained here: https://developers.google.com/storage/docs/json_api/v1/how-tos/batch

Implementing the body generator for the request wasn't too hard. Now I'm thinking about how to write a streaming response parser.

I would like to have an api similar to this:

http.request(opts, function(res) {

    var parser = new Parser()

    res.pipe(parser)

    parser.on('part', function(part) { … })

})

There's dicer[1], which powers busboy[2].

[1] https://github.com/mscdex/dicer
[2] https://github.com/mscdex/busboy

[Hendrik Cech]

Dicer looks great, thank you.
The bigger problem still remains: I need to parse the parts contents.
They are http responses themselves.

2014-04-07 16:00 GMT+12:00 mscdex <msc...@gmail.com>:
> On Sunday, April 6, 2014 9:26:24 PM UTC-4, Hendrik Cech wrote:
>>
>> I'm currently trying to make multipart/mixed requests and parse the
>> responses. The format is explained here:
>> https://developers.google.com/storage/docs/json_api/v1/how-tos/batch
>> Implementing the body generator for the request wasn't too hard. Now I'm
>> thinking about how to write a streaming response parser.
>> I would like to have an api similar to this:
>>
>> http.request(opts, function(res) {
>> var parser = new Parser()
>> res.pipe(parser)

>> parser.on('part', function(part) { ... })

>> })
>>
>
> There's dicer[1], which powers busboy[2].
>
> [1] https://github.com/mscdex/dicer
> [2] https://github.com/mscdex/busboy
>

> --
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "nodejs" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/nodejs/-IMTql2_NTU/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> nodejs+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Simeon Chaos]

Maybe you can have a look at https://github.com/chaosim/peasy, which is a simple parser, or a simple method to write a parser. peasy is not based on stream,  maybe you can write yourself a parser based on stream by using the method that peasy has demonstrated.

在 2014年4月7日星期一UTC+8下午12时22分18秒，Hendrik Cech写道：

[Floby]

Hello, this is an interesting use case.

Maybe looking at how http2 works https://www.npmjs.org/package/http2 can help you. It's basically the same principle: several responses multiplexed into a single connection. Only the framing differs (multipart vs. http2)

Alternatively, you could try and instanciate yourself a http.IncomingMessage (response object) [1] whose constructor only takes a "socket" parameter for which a readable stream should suffice. You may have to trick the IncomingMessage into thinking the stream is a socket.

I'd be interested in knowing if that works out.

[1] https://github.com/joyent/node/blob/master/lib/_http_incoming.js

[Floby]

I looked into this a bit further.

Unfortunately busboy won't help you as it only handles multipart/form-data bodies.

However dicer could easily do the trick for parsing requests.

However, if you need to generate a payload against this API, maybe partly [1] can help you.

https://www.npmjs.org/package/partly

[Hendrik Cech]

Yep, dicer looks great for getting the response parts.

Thanks, I will take a look at partly.

Unfortunately the code needs to run in the browser as well. I can't make use of the http module therefore.

For reference, this would be a good starting point otherwise: https://gist.github.com/hendrikcech/10101165

It seems like I will have to hack something together myself.

[mscdex]

On Tuesday, April 8, 2014 4:14:34 AM UTC-4, Hendrik Cech wrote:

Unfortunately the code needs to run in the browser as well. I can't make use of the http module therefore.

You may want to check how various browsers handle a multipart/mixed response first. Some may not give you the entire response, some may just give you the last part, others may do something entirely different.

[Hendrik Cech]

You may want to check how various browsers handle a multipart/mixed response first. Some may not give you the entire response, some may just give you the last part, others may do something entirely different.

The latest versions of Chrome, Firefox and Safari on OS X are looking good fortunately. They give the entire response.

[Joe David]

You can use a baisc http parser like this: https://www.npmjs.com/package/http-message-parser