Bug in CAtlHttpClientT::ReadChunkedBody() ?

[Ravi K.]

I've been considering using the CAtlHttpClient class for my application, and
installed tried out the HttpClient sample application.

I'm finding that when I click the 'navigate' button having set the url to a
server that sends back chunked responses, the sample application does not
read the body of the response, and gets into an error state. (Try out a url
such as www.yahoo.com for the sample app. The server returns a chunked
response).

I debugged it a little bit and it did seem that

CAtlHttpClientT<TSocketClass>::ReadChunkedBody()

was not parsing and using the size field in the chunked response. Any one
else run into this issue ?

Thanks for any pointers,
-Ravi

[Wei Mao [MS]]

Ravi,
I have a look at CAtlHttpClientT<TSocketClass>::ReadChunkedBody() in
\vs.net\vc7\atlmfc\include\atlhttp.inl. It properly calls
get_chunked_size() / get_chunked_data() and i do not see any error there.

After that I did some test myself: setup IIS server (ref Q278998 HOWTO
Enable Chunked Transfer Encoding with IIS); build \Microsoft Platform
Sdk\Samples\Web\iis\isapi\extensions\Chunk sample; then run HttpClient
again the server using following URL:
http://localhost/chunk/ctetest.dll?file=/chunk/Ctetest.c+chunksize=100
And it works Here is the output from the HttpClient window:

--==[[ HEADER ]]==--
HTTP/1.1 200 OK
Server: Microsoft-IIS/5.1
Date: Mon, 11 Nov 2002 07:55:53 GMT
Transfer-encoding: chunked
Content-type: text/plain
--==[[ BODY ]]==--
/*++

Copyright (c) 1997 - 2000 Microsoft Corporation
.. omitted ...
*buf = '\0';
return TRUE;
}

Then I did some debugging. it also proves that ReadChunkedBody works as
exepcted. therefore it may not be the problem of CAtlHttpClientT
implementation here.

thanks
wei

--
This posting is provided "AS IS" with no warranties, and confers no rights.
Got .Net? http://www.gotdotnet.com

[Ravi K.]

Thanks for testing it out, Wei.

I wonder if the bug is related to chunk sizes - i.e. if you point the
HttpClient sample app url to www.yahoo.com, for example, the chunk sizes are
28K or so. There seems to be in initial chunk buffer size of 2048 in
atlhttp.inl. ( #define CHUNK_BUFF_SIZE 2048).

Maybe the thing to do is to then try the sample app with larger chunk sizes
(say 28K) (or try out you url test with chunksize = 28000 or something large
like that).

Thanks again,
-Ravi

"Wei Mao [MS]" <wei...@online.microsoft.com> wrote in message
news:vm3zk4ViCHA.2476@cpmsftngxa06...

[Wei Mao [MS]]

Ravi,
I test chunksize=30000 and still do not meet any problem. as to
CHUNK_BUFF_SIZE, the comment says:

// Keeps track of the allocated size of t_chunk_buffer.
// This size will change if we need to read chunks bigger
// than the currently allocated size of t_chunk_buffer.
long nTChunkBuffSize = CHUNK_BUFF_SIZE;

it is not likely a problem too.

[Ravi K.]

Thanks for you patience Wie.

I decided to debug what the issue was with the chunked data from
www.yahoo.com.

What I found was the yahoo web server is inserting spaces (0x20) characters
between the chunk size hex value and the 0x0d0x0a (\r\n) characters.

I modified get_chunked_size() (see the code below) to ignore 'spaces'
following the end of the hex digits, and then everything worked fine.

The question is then if the chunked data is in violation of RFC2616. I need
to look closer at the spec, but I don't think LWS (line white space)
separating the hex size token from the rest of the message body is violating
the standard. So maybe worth making the atl code a bit defensive for this
situation ?...

-Ravi

template<class TSocketClass>

inline CAtlHttpClientT<TSocketClass>::CHUNK_LEX_RESULT
CAtlHttpClientT<TSocketClass>::get_chunked_size(char *&pBuffStart, char
*&pBuffEnd, long* pnChunkSize)

{

CHUNK_LEX_RESULT result = LEX_ERROR;

char *pStop = NULL;

if (pBuffStart >= pBuffEnd)

result = LEX_OUTOFDATA;

else

{

long nResult = strtoul(pBuffStart, &pStop, 16);

while (*pStop == ' ') <=== (added)

pStop++; <=== (added)

"Wei Mao [MS]" <wei...@online.microsoft.com> wrote in message

news:lJDXhTgiCHA.2684@cpmsftngxa06...

[Wei Mao [MS]]

Ravi,
It is good to know that you've got the root cause of the problem. Actually
I've tried to connect www.yahoo.com before; however since I am behind the
corporation firewall, I cannot see the chunking at all (per rfc2616
sec19.4.6); that is why I turn to an isapi sample for testing.

I will convey your concern about the extra space character to our enginers
too since I am not familiar with this kind of details.

[Sam Thomsen]

I am also getting some strange behavior with CAtlHttpClientT when
downloading chunked data. There are no errors reported when calling
Navigate() and the value returned by GetBodyLength() is correct. But
the buffer returned by GetBody() is currupt.

I checked to see if there were any strange characters in the chunk
size and didn't find anything between the size and the \r\n.

[Sam Thomsen]

I discovered the bug that was causing my problem. It is in the ATL
code.

On line 1965 in file "atlhttp.inl" you must include the length of the
buffer when appending. This is because the CAtlIsapiBuffer::Append
method uses strlen to get the buffer size if none is provided. Of
course this is not suitable for binary data.

So the fix looks like this... (Line 1965 of atlhttp.inl)

if (!m_current.Append(( LPCSTR )result_buffer,
result_buffer.GetLength() ))

Another problem was found on line 1596 of the same file where memcpy
was used to do an overlapped memory move. memmove should be used
instead. memcpy may have been inappropriately used in other places
too.

Sam Thomsen