WinHttpSendRequest - total data limit

[Shuo Shen]

Hello,

I am considering sending hugh data transfer for certain big companies over
http. The data size can easily reach 10G, 20G etc.

The API WinHttpSendRequest has only a DWORD data structure for
dwTotalLength, which is about 4G. How can I do more?

Microsoft's file size structure can reach GG size level(of course, I won't
do that more :), why does the new WinHTTP API seem to have such a low limit?

This should be easily corrected if it's an issue, right?

Thanks,
Shuo

[Shuo Shen]

[Stephen Sulzer (Microsoft)]

Unfortunately, this is a limitation in WinHTTP currently. I believe the
limit that WinHTTP supports is actually 2 gigabytes, not 4GB.

Regards,

Stephen Sulzer
Microsoft Corporation

"Shuo Shen" <no_spa...@softartisans.com> wrote in message
news:#cHKbM7PBHA.1448@tkmsftngp05...

[Bob Landau]

Shuo,

WinHttp breaks down the buffer to M sized packets and passes this to
Winsock. This in turn will pass the buffer directly down to the driver so
must have enough non-paged pool RAM to do this. Lastly this is further
broken down at the IP to the MSS which is some what less than 1500 bytes
for ethernet.

I guess this is a long winded way of saying you will not gain any speed
and I suspect the throughput on the machine will be slower by passing huge
amounts of data at one time; since the M gigabyte buffer will be locked
down while you're filling it rather than allowing other processes or the OS
to use this memory.

Rather than do that one should be calling WinHttpWriteData multiple times.
This is a fast call call that does passing the data off to the completion
port threads and returns. There is no limit on the amount of data that can
be sent in this way.

Lastly and I'm assuming here, you are speaking about the IA64 port of
WinHttp given that any 32 bit processor is only able to address 2^32 4G
bytes of memory. This discrepency was pointed out before whether this will
be changed in the next release is not known however I will pass on your
comments to the dev group.

thx
bob

This posting is provided “AS IS” with no warranties, and confers no rights.
You assume all risk for your use. © 2001 Microsoft Corporation. All rights
reserved.

--------------------
From: "Shuo Shen" <no_spa...@softartisans.com>
Subject: WinHttpSendRequest - total data limit
Date: Mon, 17 Sep 2001 15:55:54 -0400
Lines: 19
X-Priority: 3
X-MSMail-Priority: Normal
X-Newsreader: Microsoft Outlook Express 5.00.2919.6600
X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6600
Message-ID: <u5#ebM7PBHA.1448@tkmsftngp05>
Newsgroups: microsoft.public.winhttp
NNTP-Posting-Host: mail.softartisans.com 63.103.15.68
Path: cppssbbsa01.microsoft.com!tkmsftngp01!tkmsftngp05
Xref: cppssbbsa01.microsoft.com microsoft.public.winhttp:323
X-Tomcat-NG: microsoft.public.winhttp

[Shuo Shen]

Thanks Stephen and Bob. I tried something different and would like to know
your opinions.

For Bob, yes, I send the data chunk by chunk by using "WinHttpWriteData",
the limit for me is the http header "Content-Length".

Stephen, as explained by Bob, your team has done many things to optimize the
transfer. Assume there is no change needed for your calls to
WinHttpWriteData, why couldn't your team change the implementation for the
header of "Content-Length". Literally, you just need to change the
dwTotalLength data type, and convert it to a string and send it with
"Content-Length".

I tried in a different way, I set "Content-Length" by using
WinHttpAddRequestHeaders, setting it to 2400000000(2.23G), I sent this
amount of data through WinHttpWriteData and it worked. I don't know this is
an unsupported case or a bug or another correct way to do this, I like it
because it works. I would like know if this is valid and if your lower level
transfer is able to handle even higher load.

Thanks,
Shuo

"Stephen Sulzer (Microsoft)" <anti-sp...@microsoft.com> wrote in message
news:#Ac4Ry7PBHA.1544@tkmsftngp05...

[Stephen Sulzer (Microsoft)]

Well, you may have found a simple work-around. If you specify more than 2GB
of total request data, WinHTTP does not form the Content-Length header
correctly, which is a bug. But if you set the Content-Length header
yourself, WinHTTP will not override it.

Since the OptionalLength and TotalLength parameters are DWORDs, WinHTTP
officially supports only up to 4GB data transfers. Whether WinHTTP works
reliably with transfers > 4GB (in all cases, using the workaround to set the
Content-Length header) is unknown. It is not a tested or supported scenario.

Regards,

Stephen Sulzer
Microsoft Corporation

"Shuo Shen" <no_spa...@softartisans.com> wrote in message

news:efYnOs9PBHA.692@tkmsftngp04...