xhr-streaming clarification
Stephen Diehl
Vladimir Dronnikov
> reading the first 10240 bytes of the stream regardless of the chunking,
I believe it's _just_ for matters of testing.
--Vladimir
Marek Majkowski
> I'm working a on a gevent server
Cool!
> implementation for sockjs and am having
> some difficulty with the xhr-streaming transport.
>
> As I understand it xhr-streaming does revolves around chunked transfer
> encoding with the total stream capped at 128k. This seems simple enough to
> implement but the unit tests seem to fail when reading the chunks.
In order to allow browser to GC the XHR connection it must be terminated
at some point by the server. There is a limit, by default 128KiB of data
that can be sent to the single http request.
But. But /echo service is required to have this limit reduced, AFAIR to
4KiB, but please do check the sockjs-node/examples/test_server code.
I hope that makes it a bit clearer.
> It appears that the r.read() in the XhrStreaming.test_transport test ends up
> reading the first 10240 bytes of the stream regardless of the chunking,
> wheras the unit test appears to expect that it will only read the first
> 10240 bytes only of the first chunk, and the subsequent calls to read() will
> then read from the second chunk. Instead the observed behvaior is that the
> httplib seems seems to be abstracting the chunking away and returning the
> concatenation of the first and second chunk at the same time thus resulting
> is the test failing with an assertion that "hhhhhhh....h\o\n"
> != "hhhhhhh....h\n" .
>
> Not sure if this is a problem with httplib ( I'm on Python 2.7 ) or the unit
> tests in general. For example, curl seems to have the right behavior.
>
> FWIW, there seem to be other problems with this test on the node
> backend, XhrStreaming.test_transport seems to just timeout.
Generally, I try to make sure that sockjs-protocol test works against
current sockjs-node (dev branch). Last time I checked it worked fine :)
Cheers,
Marek
Marek Majkowski
I think I misunderstood the question, let me try again :)
On Sat, Jan 28, 2012 at 17:06, Stephen Diehl <stephen...@gmail.com> wrote:
> I'm working a on a gevent server implementation for sockjs and am having
> some difficulty with the xhr-streaming transport.
>
> As I understand it xhr-streaming does revolves around chunked transfer
> encoding with the total stream capped at 128k. This seems simple enough to
> implement but the unit tests seem to fail when reading the chunks.
>
> It appears that the r.read() in the XhrStreaming.test_transport test ends up
> reading the first 10240 bytes of the stream regardless of the chunking,
> wheras the unit test appears to expect that it will only read the first
> 10240 bytes only of the first chunk, and the subsequent calls to read() will
> then read from the second chunk.
This particular line is confusing:
https://github.com/sockjs/sockjs-protocol/blob/master/utils.py#L65
The flow then goes here:
https://github.com/sockjs/sockjs-protocol/blob/master/httplib.py#L568
Basically, the expectation is that server is using HTTP/1.1 chunking.
The tests indeed hang if your server doesn't do chunking.
This is often confusing, we have an issue opened to clarify this:
https://github.com/sockjs/sockjs-protocol/issues/26
> Instead the observed behvaior is that the
> httplib seems seems to be abstracting the chunking away and returning the
> concatenation of the first and second chunk at the same time thus resulting
> is the test failing with an assertion that "hhhhhhh....h\o\n"
> != "hhhhhhh....h\n" .
>
> Not sure if this is a problem with httplib ( I'm on Python 2.7 ) or the unit
> tests in general. For example, curl seems to have the right behavior.
Sockjs-protocol testsuite expects http transfer-encoding: chunked.
> FWIW, there seem to be other problems with this test on the node
> backend, XhrStreaming.test_transport seems to just timeout.
>
> Thanks in advance!
Marek
Marek Majkowski
>> It appears that the r.read() in the XhrStreaming.test_transport test ends up
>> reading the first 10240 bytes of the stream regardless of the chunking,
>> wheras the unit test appears to expect that it will only read the first
>> 10240 bytes only of the first chunk, and the subsequent calls to read() will
>> then read from the second chunk.
>
> This particular line is confusing:
> https://github.com/sockjs/sockjs-protocol/blob/master/utils.py#L65
>
> The flow then goes here:
> https://github.com/sockjs/sockjs-protocol/blob/master/httplib.py#L568
>
> Basically, the expectation is that server is using HTTP/1.1 chunking.
> The tests indeed hang if your server doesn't do chunking.
>
> This is often confusing, we have an issue opened to clarify this:
> https://github.com/sockjs/sockjs-protocol/issues/26
>
I committed initial work for #26, please take a look:
./venv/bin/python sockjs-protocol-dev.py Http10
./venv/bin/python sockjs-protocol-dev.py Http11
Cheers,
Marek