InternetReadFile much slower than IE
meanfox
Hi,
When I use IE to download a test file it takes about 3 seconds (non-
cached).
My own WinInet code using InternetReadFile takes about 7 seconds (also
non-cached).
This is repeatable. It is a 644K file. This is on XP SP2 with IE7.
Can anyone explain this and tell me how to speed up my WinInet
download? I thought IE used WinInet?
I have experimented with a few things that made no difference:
* Running synchronously vs running in a thread
* Downloading in 1 call to InternetReadFile vs. downloading many 1K
chunks
Please advise!
Tom
Scherbina Vladimir
--
--Vladimir, Windows SDK MVP
"meanfox" <code...@gmail.com> wrote in message
news:1174544116....@y66g2000hsf.googlegroups.com...
meanfox
Here's a minimal test app that demonstrates the problem.
Right now I am getting around 15-25 seconds for the WinInet download!
IE7 is taking 3-4s for the same file.
I need the INTERNET_FLAG_PRAGMA_NOCACHE because without it I get
caching errors. Another can of worms.
What am I doing wrong?
Thanks -
Tom
#include <windows.h>
#include <wininet.h>
#include <assert.h>
#include <stdio.h>
#include <conio.h>
void main(void)
{
HINTERNET session = ::InternetOpen("wininet_test",
INTERNET_OPEN_TYPE_PRECONFIG, NULL, NULL, 0);
assert(session != NULL);
HINTERNET h_http_file = InternetOpenUrl(session, "http://
www.fundictive.com/gamepatch/client_r.exe", NULL, 0,
INTERNET_FLAG_PRAGMA_NOCACHE, 0);
assert(h_http_file != NULL);
char szSizeBuffer[32] = "";
DWORD dwLengthSizeBuffer = sizeof(szSizeBuffer);
BOOL bQuery = ::HttpQueryInfo(h_http_file,
HTTP_QUERY_CONTENT_LENGTH, szSizeBuffer, &dwLengthSizeBuffer, NULL);
assert(bQuery == TRUE);
DWORD dwFileSize = atol(szSizeBuffer);
char * pdata = new char[dwFileSize];
DWORD length = 0;
DWORD time_start_ms = timeGetTime();
BOOL bRead = ::InternetReadFile(h_http_file, pdata, dwFileSize,
&length);
printf("InternetReadFile took %d ms\n", timeGetTime() -
time_start_ms);
assert(length == dwFileSize);
BOOL rc_ich = ::InternetCloseHandle(h_http_file);
assert(rc_ich == TRUE);
rc_ich = ::InternetCloseHandle(session);
assert(rc_ich == TRUE);
FILE * file = ::fopen("client_r.exe", "wb");
::fwrite(pdata, 1, length, file);
::fclose(file);
delete [] pdata;
getch();
}
Paul Baker [MVP, Windows - Networking]
getting it from the origin server.
Internet Explorer may be getting it from the proxy cache or from its own
cache, depending on settings. This would of course be faster.
Paul
"meanfox" <code...@gmail.com> wrote in message
news:1174567755.9...@p15g2000hsd.googlegroups.com...
meanfox
Oh I thought I had stopped IE7 from doing any caching at all. Is there
actually a way to do that so that I can compare IE7 properly with my
WinInet app?
I turned on INTERNET_FLAG_PRAGMA_NOCACHE because when I updated the
files by uploading them, neither IE nor my WinInet application
detected that the files had changed for some time period. I think some
small text files (my manifests) were detected correctly, but my exe's
were not. Is that expected? How should I resolve that? My manifest
implied that the files had been updated, but the fetch retrieved stale
files...
Tom
Paul Baker [MVP, Windows - Networking]
INTERNET_FLAG_PRAGMA_NOCACHE flag offers comparable performance. This should
give you the same caching behaviour as Internet Explorer.
The behaviour you describe may simply be the behaviour of the proxy. If
you're always forcing it to load the page from the origin server, it's could
be slower. If you're not, the proxy might not give you a fresh copy.
There are a lot of things going on here that I cannot claim to understand
fully, so let's take it a step at a time.
Paul
"meanfox" <code...@gmail.com> wrote in message
news:1174570214.7...@p15g2000hsd.googlegroups.com...
meanfox
I appreciate your help.
Omitting INTERNET_FLAG_PRAGMA_NOCACHE does indeed speed it up heaps
but timings are still poorer than IE:
In ms:
8375
3125
4282
6140
11859
5265
As opposed to IE which is 2-4 s every time.
Tom
Paul Baker [MVP, Windows - Networking]
InternetReadFile, but I don't know if I have any real reason for that
number. I do know that 32 bytes just seems like it is too small.
How big is the file you are testing with?
If you have Internet Explorer 4 or higher, you could look at using
InternetReadFileEx. I have not used this function, but my impression is that
it would not help performance. In this thread, Microsoft Developer Support
suggests that "There is not really any advantages to using either one":
Paul
"meanfox" <code...@gmail.com> wrote in message
news:1174576584.2...@e65g2000hsc.googlegroups.com...
meanfox
Changing the buffer size to 1K helped marginally if at all. The file
is 644K.
I tried the InternetReadFileEx code but saw no improvement. (BTW that
code needs a memset for InetBuffers).
I wonder if tcpip would be faster...
I guess we can just conclude that INTERNET_FLAG_PRAGMA_NOCACHE slows
things down either significantly or drastically, but solves cacheing
errors.
Thanks again Paul...
Tom
Paul Baker [MVP, Windows - Networking]
, it was slower than Internet Explorer.
This is unsolved mystery, because Internet Explorer uses WinInet just like
you.
Paul
"meanfox" <code...@gmail.com> wrote in message
news:1174621121.5...@e1g2000hsg.googlegroups.com...
meanfox
It is unsolved yes.
But now I am more worried about how slow it can be if I use
INTERNET_FLAG_PRAGMA_NOCACHE, which I seem to need to use.
Is WinInet the best option I have?
My game will check a manifest for updates every time it runs.
Tom
Paul Baker [MVP, Windows - Networking]
Whatever software you use, you will need to decide what to do about a
situation where you have a proxy, which apparently you do in your network.
If you use INTERNET_FLAG_PRAGMA_NOCACHE, I am wondering if WinInet is
checking the modified date of the file according to the origin server and
download it if it is newer than it cached, or if just downloads it
regardless. This may depend on Internet Options. I do not claim to know the
answers to these questions. You can use tools such as Fiddler
(http://fiddlertool.com/fiddler/).
If the problem is that WinInet is downloading it regardless, you can check
pass HTTP_QUERY_LAST_MODIFIED to HttpQueryInfo to get the modified time and
decide for yourself whether or not to download it.
You can of course try using third party software or building on top of lower
level software like WinSock, that will certainly give you more control. But
that sounds like more work to me.
Paul
"meanfox" <code...@gmail.com> wrote in message
news:1174662729.8...@l75g2000hse.googlegroups.com...
Scherbina Vladimir
--
--Vladimir, Windows SDK MVP
"meanfox" <code...@gmail.com> wrote in message
news:1174662729.8...@l75g2000hse.googlegroups.com...
meanfox
Paul,
Yes I'm sure there is a proxy at my ISP; and
INTERNET_FLAG_PRAGMA_NOCACHE certainly seems to fix the stale files
problem. It appears not to interfere with local timestamp-based
cacheing because I can see the local cache also being used, and reset
when I clear the temporary internet files in IE. If I do see any stale-
file problems I will add INTERNET_FLAG_RELOAD and possibly other
flags.
Vladimir,
I elected WinInet over WinHTTP because from what I read WinInet seemed
the appropriate API. My code is doing autoupdate for a game client. Do
you think I should give WinHTTP a go?
Cheers,
Tom
Paul Baker [MVP, Windows - Networking]
Yes, I should have mentioned INTERNET_FLAG_RELOAD. It will certainly prevent
it from looking in the local cache, and I would expect that a "Pragma:
no-cache" header is also added in this case, preventing it from looking in
the proxy cache. You can verify that using Fiddler.
Then you can just look at the modified time and decide whether the file
changed yourself. You might consider using a HEAD request. If you do that,
you will get only the headers and will have to submit another, GET, request
if you actually want the file. So it's more efficient on both client and
server when there is not a newer version but less efficient if there is.
Probably not a big deal either way ;)
If this works for you, I see no reason to rewrite everything WinHTTP. It
depends on how much you've invested in WinInet already. WinHTTP is more
service oriented and will not try to keep a cache, use Internet Options or
otherwise act like Internet Explorer. This might be beneficial.
Paul
"meanfox" <code...@gmail.com> wrote in message
news:1175053888.7...@l77g2000hsb.googlegroups.com...
Paul Baker [MVP, Windows - Networking]
You might think I'm helping *you*, but actually you helped *me* too :)
I have something similar to this and I am not using any flags. I happen not
to be going through a proxy and my caching behaviour seems to be okay, so
it's not a big deal. But I should be using INTERNET_FLAG_RELOAD. I verified
using Fiddler that this flag does cause there to be a "Pragma: no-cache"
header in the request, even in the absence of INTERNET_FLAG_PRAGMA_NOCACHE.
RFC 3143 Known HTTP Proxy/Caching Problems which may be of interest:
http://tools.ietf.org/html/rfc3143#section-2.2.1
It implies that if there is a proxy, you would expect a "Cache-control:
no-cache" or "Cache-control: must-revalidate" header in the response. So
this can presumably be used to determine whether or not there is proxy
involvement.
Paul
"Paul Baker [MVP, Windows - Networking]" <pa...@online.rochester.rr.com>
wrote in message news:uiaEZ4T...@TK2MSFTNGP03.phx.gbl...
meanfox
Thanks Paul,
I may tinker with it some more at a later date. At least now I have a
better understanding of what is going on. The rfc link you gave looks
useful.
Normally there will be no update for my game.
Tom
Paul Baker [MVP, Windows - SDK]
Paul
"meanfox" <code...@gmail.com> wrote in message
news:1175409145.5...@n76g2000hsh.googlegroups.com...