Error streaming long http output

[Jonathan Ragan-Kelley]

I built a lightweight http server which streams (large binary) output
from subprocesses as binary data through the http response. I have a
stripped-down example here:

http://gist.github.com/571296

Many clients (Chrome, curl) return errors mid-way through downloading
the stream, at seemingly random (and variable) points, from a few
hundred kb to at most a few tens of MB into the transaction. The curl
--trace gives little immediately obvious reason for the failure.

Salient points:

- this happens for multiple types of subprocess and multiple types of
streamed binary data.
- file size is not known a priori, so is not set in the header.

Is this a node http bug, am I just doing something illegal, or what?

Many thanks.

(Aside: node is an outstanding way to write this sort of server. Thank
you.)

[mscdex]

What about setting Content-Length? Also, instead of executing cat, you
could open a readstream for the file and use sys.pump to write it to
the response stream.

[mscdex]

I've forked your gist here with the suggested changes I mentioned:
http://gist.github.com/571343

[Jonathan Ragan-Kelley]

To be clear, cat isn't the point. I would not have used spawn('cat')
just to read a file. That's just the simplest subprocess to use in
this example. In reality I'm generating compressed streams (e.g. zip).

And no, I'm not setting content length, because it is not known ahead
of time. That is intentional. In my understanding it is optional, and
only required as a hint to aid e.g. download manager GUIs in
displaying useful progress bars, not for the correctness of the actual
transaction.

Since I forgot to specify in the original message (it is in the log in
that gist), curl errors out with:

curl: (56) Received problem 3 in the chunky parser

[Ben Noordhuis]

You need to set 'Transfer-Encoding: chunked' and omit the Content-Length header.

[mscdex]

On Sep 9, 4:48 am, Ben Noordhuis <i...@bnoordhuis.nl> wrote:
> You need to set 'Transfer-Encoding: chunked' and omit the Content-Length header.

That header should be added automatically if Content-Length isn't set.

[hoodoos]

Try not writing to /dev/null and see what is wrong with chunk transfer
in the end. It can send wrong number of bytes in chunk header or
something. Just catch last 10-20 kilobytes. Are you familiar with
chunk transfer in common?

Here's some info:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html 3.6.1
http://en.wikipedia.org/wiki/Chunked_transfer_encoding

You can also try to write a tcp client on node, to better see what
happen with chunks, but you'll need to write a parser to monitor
correct chucks sizes. :)

[r...@tinyclouds.org]

Yes - this is a bug. I'm not sure immediately what the problem is.
I'll get back to you soon.

Thanks for the report.

[Jonathan Ragan-Kelley]

Thanks for the confirmation, Ryan. Any word on this? Is there a bug I
should be following? I need to either resolve or work around this to
launch a feature.

On Sep 9, 9:26 pm, r...@tinyclouds.org wrote:
> On Wed, Sep 8, 2010 at 8:16 PM, Jonathan Ragan-Kelley
>

[Jonathan Ragan-Kelley]

[bump]

The bug persists in 0.2.3. Any word? I'd really like not to have to
move to Event Machine or Twisted just because of HTTP encoder trouble.
Just having some visibility into whether/when this might get looked at
more deeply would help us plan.

Thanks again.

[Nathan Rajlich]

Have you tried disabling Node's chunked encoding by setting a response header "Transfer-Encoding: identity"?

--
You received this message because you are subscribed to the Google Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com.
To unsubscribe from this group, send email to nodejs+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nodejs?hl=en.

[Matt Ranney]

On Wed, Oct 20, 2010 at 12:44 PM, Jonathan Ragan-Kelley <kato...@gmail.com> wrote:

The bug persists in 0.2.3. Any word? I'd really like not to have to
move to Event Machine or Twisted just because of HTTP encoder trouble.
Just having some visibility into whether/when this might get looked at
more deeply would help us plan.

Have you tried without specifying an encoding?  Instead of:

        res.write(data, "binary");

How about:

        res.write(data);

[Ryan Dahl]

I've done some debugging of this - it does appear to be a bug in the parser.

[Jonathan Ragan-Kelley]

This appears fixed in the 0.3.0 unstable drop (though not 0.2.4).
Thanks.

On Oct 22, 4:46 pm, Ryan Dahl <r...@tinyclouds.org> wrote:
> On Fri, Oct 22, 2010 at 12:21 PM, Matt Ranney <m...@ranney.com> wrote:

> > On Wed, Oct 20, 2010 at 12:44 PM, Jonathan Ragan-Kelley <katok...@gmail.com>

[Ryan Dahl]

On Mon, Oct 25, 2010 at 6:07 PM, Jonathan Ragan-Kelley
<kato...@gmail.com> wrote:
> This appears fixed in the 0.3.0 unstable drop (though not 0.2.4).
> Thanks.

Thanks.

The commit that fixed it was 7e24a05cba2373c0ab1f0629366b664a593f357f.
I backported it to v0.2 in c42a5cc8eb6d489408c4f86995fdbdc27943fe96.
I'll release v0.2.5 in a few days.