Problems with WinInet caching (and not sending) data
Judy Booth
My problem is that all but the last block of data seems to get through
to the web server I'm using for testing and I can't work out how to
flush out the last bit.
I'm using HttpAddRequestHeaders to set the following headers for my
data:
- Content-Type: application/octet-stream;
- Content-Transfer-Encoding: binary
Without these the data seems to be treated as something like text and
I get a spurious CR-LF at the end of my data that I don't want. Also
without "binary" data specified I get the chunks of data through to
the server in the same blocksizes I used for the POSTs.
With the "binary" headers set the data seems to come through in 128
byte chunks to the server, but the last block never gets there, even
though I have closed the handles and my client app has terminated.
I have tried using both HttpSendRequest and HttpSendRequestEx with
InternetWriteFile and HttpEndRequest and it doesn't make any
difference.
I'm also using HttpQueryInfo to confirm that the server status is OK.
If anyone has any ideas about what I may be missing, or not setting in
the flags, then I'd be grateful for any suggestions.
Thanks
Judy Booth
P.S. please reply to the newsgroup and not to the email address on
this message.
Brian Combs
You sould get a network trace or use the debug build of WinInet to see what
it is sending. You can also check the IIS log to see how much data was sent
with your request. WinInet should set a content-length header that you can
use to varify with network trace to make sure all of the data went. Most
Web Servers will give some kind of error if data sent does not match
content-length header.
for debug build of wininet.dll
ftp://ftp.microsoft.com/PSS/Tools/Developer%20Support%20Tools/WinInet/
Thanks
Brian [MS]
Microsoft Developer Support
This posting is provided "AS IS" with no warranties, and confers no rights.
--------------------
| Newsgroups: microsoft.public.inetsdk.programming.wininet
| Subject: Problems with WinInet caching (and not sending) data
| From: Judy Booth <judy.nos...@nospam.dial.pipex.com>
| Message-ID: <Xns941F9BCFAC84B...@195.129.110.201>
| User-Agent: Xnews/06.08.25
| Date: 25 Oct 2003 14:19:01 GMT
| Lines: 32
| NNTP-Posting-Date: 25 Oct 2003 14:19:01 GMT
| NNTP-Posting-Host: 62-241-186-175.dsl.pipex.com
| X-Trace: 1067091541 news.dial.pipex.com 5658 62.241.186.175:3046
| X-Complaints-To: ab...@uk.uu.net
| Path:
cpmsftngxa06.phx.gbl!TK2MSFTNGP08.phx.gbl!newsfeed00.sul.t-online.de!t-onlin
e.de!skynet.be!skynet.be!newsgate.cistron.nl!news.cambrium.nl!news.cambrium.
nl!news.cambrium.nl!newsfeed.kabelfoon.nl!195.129.110.21.MISMATCH!bnewsfeed0
0.bru.ops.eu.uu.net!bnewsinpeer01.bru.ops.eu.uu.net!lnewsinpeer01.lnd.ops.eu
.uu.net!lnewspost00.lnd.ops.eu.uu.net!emea.uu.net!not-for-mail
| Xref: cpmsftngxa06.phx.gbl
microsoft.public.inetsdk.programming.wininet:10429
| X-Tomcat-NG: microsoft.public.inetsdk.programming.wininet
Judy Booth
Thank you for the information about using the debug version of
WinInet. Infuriatingly when I re-tried this with the debug build my
"caching" problem had gone away! I've been trying other things with
WinHTTP since my post and my guess is that something I've tweaked while
doing that has "cured" the problem.
If the original problem does come back then I will post an update here
with what I can find out about what was causing the problems.
However, your information about how to get trace info from WinInet is
very useful for the rest of what I am doing.
Thanks again,
Judy Booth
Br...@online.microsoft.com (Brian Combs) wrote in
news:n7yKE#VnDH...@cpmsftngxa06.phx.gbl:
> ps.eu uu.net!lnewspost00.lnd.ops.eu.uu.net!emea.uu.net!not-for-mail
Hector Santos
data. Some web servers do not support it or don't read it in after reading
in the content-length, and that is when you can run into trouble with recent
IE updates.
It is possible that recent versions of WININET.DLL was updated to support
the extra CRLF, hence explaining why many web servers and IE browser
operations are experiencing "page cannot be displayed" and/or URL resends,
both of which are related.
In fact, microsoft is offering a hoxfix for WININET.DLL that you have to
call to get from them:
http://support.microsoft.com/default.aspx?scid=kb;en-us;818139
Basically, this is the problem as I see it after 2-3 week of working on the
problem.
Because of the extra bytes still remaining in the read queue, a socket reset
occurs at some point that is driven recent IE versions nuts. It is
completely intermittent depending on transmission speed in both direction,
caching and the size of the TCP/IP window.
When the socket is closed by the server with bytes remaining in the receive
queue and IE is still reading response data from the server, a RST is
received by the browser that is causing IE show one of the two behaviors
indicated above.
Traditionally, hosting Servers use SO_LINGER on socket so that the
closesocket command is blocked until all the data is sent by the server.
The FIN signal is not sent until all the data is sent and acknowledged by
the receiver.
This is where the network transmissions come into play which makes it
intermittent.
Something has come wrong with the SO_LINGER support. If there is still
data in the receive queue the server has not read in, it seems that a RST
is sent instead of a FIN. The IE browser sees this RST and results in
one of the two behaviors depending on the user's "Show Friendly HTTP
response" setting.
The solution is:
1) Adjust the web server to handle the extra bytes in the posted data, and
2) Use TCP Half Close methods to terminate the connection as suggested by
MSDN and the book TCP/IP Illustrated Volume 1 (Page 238).
TCP Half Close uses the shutdown() command and a flush of the receive queue
as so:
shutdown(socket);
while (recv(socket,buffer,sizeof(buffer)) > 0);
closesocket(socket)
This is technically the proper way the server and client should gracefully
terminate a socket. However, as the TCP/IP iIlustrated book says, it is
not widely supported.
The shutdown() tells the receiver that NO MORE DATA will be sent by the
sender. It does not close the socket. It tells the receiver to CLOSE its
socket once it is finished reading data.
The host waits for the remote FIN signal by using recv() in a loop.
The recv() loop will exit when the socket is closed.
At this state, we are "half close."
Finally, the closesocket() closes the second half of the socket connection
and we have a graceful termination.
All this came about after 2-3 weeks of having all these recent IE issues
with customer causing me do investigate the strange problem. The above is
the resultant summary work I did.
I still believe Microsoft screwed up the browser some how. It may be that
they added this extra CRLF pair to comply with suggested standards for the
POST command. Don't know. But they have come up with this HotFix that
you have to call to get. Which tells me they are still investigating it.
Hope it helps
--
Hector Santos
WINSERVER "Wildcat! Interactive Net Server"
support: http://www.winserver.com
sales: http://www.santronics.com
"Judy Booth" <judy.nos...@nospam.dial.pipex.com> wrote in message
news:Xns941F9BCFAC84B...@195.129.110.201...
Brian Combs
RFC for HTTP (http://www.ietf.org/rfc/rfc2616.txt) does not require extra
CRLF at the end of entity data that is posted to web server. In fact it
states that it is forbidden. This was only done on some clients for HTTP
1.0 and not for HTTP 1.1. it was done to try and keep the connection alive.
HTTP 1.1 added keep alive support so this 1.0 hack is not needed. In fact
it can cause a web server to reset the connection. See
816405 BUG: Microsoft Internet Explorer Displays a Blank Page While It Uses
an
http://support.microsoft.com/?id=816405
The hotfix you gave has nothing to do with posting data but fixes a
different problem. In fact most new webservers will reset connection for
security reasons if Post data is not correct. That is incorrect
Content-Length header or incorrect chunked data if you are using chunked
for upload.
Thanks
Brian [MS]
Microsoft Developer Support
This posting is provided "AS IS" with no warranties, and confers no rights.
--------------------
| From: "Hector Santos" <nos...@nospam.com>
| References: <Xns941F9BCFAC84B...@195.129.110.201>
| Subject: Re: Problems with WinInet caching (and not sending) data
| Date: Sat, 15 Nov 2003 00:27:11 -0500
| Lines: 133
| MIME-Version: 1.0
| Content-Type: text/plain;
| charset="Windows-1252"
| Content-Transfer-Encoding: 7bit
| X-Priority: 3
| X-MSMail-Priority: Normal
| X-Newsreader: Microsoft Outlook Express 6.00.2800.1158
| X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1165
| Message-ID: <#BIEUkzq...@TK2MSFTNGP11.phx.gbl>
| Newsgroups: microsoft.public.inetsdk.programming.wininet
| NNTP-Posting-Host: adsl-153-120-163.mia.bellsouth.net 68.153.120.163
| Path: cpmsftngxa06.phx.gbl!TK2MSFTNGP08.phx.gbl!TK2MSFTNGP11.phx.gbl
| Xref: cpmsftngxa06.phx.gbl
microsoft.public.inetsdk.programming.wininet:10508
| X-Tomcat-NG: microsoft.public.inetsdk.programming.wininet
Hector Santos
"Brian Combs" <Br...@online.microsoft.com> wrote in message
news:dwFfFZ7r...@cpmsftngxa07.phx.gbl...
> Hello
> RFC for HTTP (http://www.ietf.org/rfc/rfc2616.txt) does not require extra
> CRLF at the end of entity data that is posted to web server. In fact it
> states that it is forbidden.
Correct. HTTP 1.1 states that no extra bytes in a POST.
> This was only done on some clients for HTTP
> 1.0 and not for HTTP 1.1. it was done to try and keep the connection
alive.
> HTTP 1.1 added keep alive support so this 1.0 hack is not needed. In fact
> it can cause a web server to reset the connection. See
> 816405 BUG: Microsoft Internet Explorer Displays a Blank Page While It
Uses
> an
> http://support.microsoft.com/?id=816405
However, the recents versions of IE 6.0 is sending a HTTP 1.1 POST request
with 2 extra bytes (and not NULLS, but rather CRLF). That is what started
this whole mess. It assumes HTTP 1.1 with the initial request.
Is there a reason why there is abnormal amount of Page Cannot Be Display and
"blank page" (the same error) reports over in recent weeks?
> The hotfix you gave has nothing to do with posting data but fixes a
> different problem. In fact most new webservers will reset connection for
> security reasons if Post data is not correct.
It is the TCP/IP layer that is doing the RST, not the WEB SERVER!!! In
fact, there is no RESET command at the application level to use. All you
can do is a hard closesocket().
If IIS (or any websever) do not flush those extra bytes, the SOCKET
(TCP/IP), not the server is issuing the RST. The server has nothing do to
with other than to take it into account, by flushing it or using TCP Half
Close logic as suggested by MSDN - see "Graceful Socket Termination" for the
closesocket() and shutdown() functions.
I appreciate your input.