Nonblocking send. Is it a BUG in Windows Socket API?

[dsh...@gmail.com]

Hello.

I have next problem with Windows Socket API.

Tested on Windows 2000, XP, 2003 and behaviour looks like a BUG.
Used AF_INET SOCK_STREAM socket in nonblocking mode with
WSAEventSelect().

Any call to send() or WSASend() in nonblocking mode mostly finished
with one of the next results:

1. Succeeded and all data is processed to sending when socket buffer
space is available [Documented].

1a. Succeeded and all data is processed to sending when data size is
more then socket send buffer size (SO_SNDBUF), for example - send 64Mb
[Not documented, data must be partially processed].

2. Failed with WSAEWOULDBLOCK [Documented, send() must be called
later].

3. Failed with WSAENOBUFS (for example 128Mb) [Not Documented, data
must be partially processed].

But never succeeded with only part of data processed [Not Documented,
see 3]!!!

This problem can be resolved by sending not more then 64Kb by one call
of send() or WSASend() function. See also MSDN KB201213, but it applies
to Windows 95/98/NT4 and blocking socket mode and one of resolutions is
"Use the socket in nonblocking or asynchronous mode."!!!

You can also see topic "async send() fails on very large data blocks
with nonblocking socket, appears to be treated as blocking (XP)" in
"comp.os.ms-windows.programmer.tools.winsock".

Is it a bug?
Maybe somebody have more information.

Best regards,
DS

[DSheva]

Hi.

Some additional information.

Using 64Kb limitation has bad performance in most cases.
Better resolution is using limit equal to 1 byte smaller than the
SO_SNDBUF value.
See MSDN KB823764: "Slow Performance Occurs When You Copy Data to a TCP
Server by Using a Windows Sockets API Program".

Best regards,
DS

[Alexander Nickolov]

It's an artifact of the WinSock implementation. You have to
live with it. Most applications use small send blocks, in the
order of 2K (or even less), so there's no reason you can't
do the same.

BTW, you'll get WSAEWOULDBLOCK after the send buffer
is full all the same. WSAENOBUFS means you sent more data
than can be accomodated in kernel buffers - e.g. out of
kernel memory. Note you are killing the performance of all
other applications in this (abusive) manner, since you are
tying up kernel memory.

--
=====================================
Alexander Nickolov
Microsoft MVP [VC], MCSD
email: agnic...@mvps.org
MVP VC FAQ: http://www.mvps.org/vcfaq
=====================================

<dsh...@gmail.com> wrote in message
news:1125396633.2...@z14g2000cwz.googlegroups.com...

[Arkady Frenkel]

The more memory you'll have - the more non-pageable memory you'll have for
socket buffers and you'll have different results , try that on 64bit machine
with few Tb and see the differens.
JFYI , socket buffer is 16K from W2K and newer and 8K before ( NT and 9x ) ,
so TCP stack do the best to divide and send the data but up to some limits
:)
Arkady

<dsh...@gmail.com> wrote in message
news:1125396633.2...@z14g2000cwz.googlegroups.com...

[DSheva]

Alexander, hi

Thank you for answer.

Maybe you know where I can find some information from Microsoft about
this artifact? I don't find anything about it in MSDN.

Help on send() and WSASend() say:

"... On nonblocking stream oriented sockets, the number of bytes
written can be between 1 and the requested length, depending on buffer
availability on both client and server computers. The select,
WSAAsyncSelect or WSAEventSelect functions can be used to determine
when it is possible to send more data."

If "transport buffers" is SO_SNDBUF and SO_RCVBUF - this help is
misleading.

Best regards,
DS

[DSheva]

Arkady, hi!

Arkady Frenkel wrote:
> JFYI , socket buffer is 16K from W2K and newer and 8K before ( NT and 9x ) ,
> so TCP stack do the best to divide and send the data but up to some limits
> :)

In my opinion, current behavior (non-partial buffer send) can help
minimize number of switches between kernel mode and user mode. But in
my current application design, I use the same thread for sending and
receiving data on the same socket. So if I try to send huge data block,
I can't receive any data, because send function is in progress (WSA
allocates huge kernel buffer and copies sending data to the buffer - it
can take relatively long time). In this case using small data block
gives better application behavior. But it is very important to use
blocks with size equal to SO_SNDBUF - 1; using static block limit value
gives poor network performance.

Best regards,
DS

[Waleri Todorov]

> Isn't overlapped I/O a good choice? You would need to preserve sent data
> during send operation, but no internal buffer space would bee required nor
> waiting for WSASend to complete.

Or if not using overlapped I/O (which has its traps and pitfalls) why
not use select() to know where it is "safe" to send/read

[DSheva]

Waleri, hello!

> Isn't overlapped I/O a good choice? You would need to preserve sent data
> during send operation, but no internal buffer space would bee required nor
> waiting for WSASend to complete.

Yes, I agree, overlapped I/O is the best choice for performance reason.
But now I need to optimize exist code which worked with nonblocking
sockets.

> Or if not using overlapped I/O (which has its traps and pitfalls) why
> not use select() to know where it is "safe" to send/read

I am already using nonblocking mode with WSAEventSelect(), it is more
fexible then select().

Best regards,
DS

[Alexander Nickolov]

> If "transport buffers" is SO_SNDBUF and SO_RCVBUF - this help is
> misleading.

It's not - it's the number of available kernel buffers, e.g. all the
available kernel memory can be tied up in a single send.

You won't find this documented per se anywhere - we've found
it out by experimenting with different send size patterns. Note
that the behavior does not contradict the documentation as it
is sufficiently vague.

--
=====================================
Alexander Nickolov
Microsoft MVP [VC], MCSD
email: agnic...@mvps.org
MVP VC FAQ: http://www.mvps.org/vcfaq
=====================================

"DSheva" <dsh...@gmail.com> wrote in message
news:1125478446.7...@f14g2000cwb.googlegroups.com...

[Arkady Frenkel]

If you need to use your buffers and not system , you can do that with
zero-buffering ( AFAIK that Linux term , but it work in Windows too ) , you
set size of the winsock buffer to zero ) so system will use your buffer and
not allocate system one for operations
Arkady

"DSheva" <dsh...@gmail.com> wrote in message
news:1125483990.2...@o13g2000cwo.googlegroups.com...

[Arkady Frenkel]

You can read where and how to use it in
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnwxp/html/comperfnetapppt1.asp
or read the section
"Who Manages the Buffers? " in
http://msdn.microsoft.com/msdnmag/issues/1000/winsock/default.aspx
Arkady

"Arkady Frenkel" <ark...@hotmailxdotx.com> wrote in message
news:uiwOQMsr...@TK2MSFTNGP10.phx.gbl...

[DSheva]

Arkady, hello!

Setting socket send buffer (SO_SNDBUF) to zero is a bad idea for
nonblocking mode. Because it change send() function behaviour, send()
doesn't complete until send really completes (send() work like in
blocking mode).

Best regards,
DS

[DSheva]

Alexander, hello!

Some interesting information from MSDN article "Windows Sockets 2.0:
Write Scalable Winsock Apps Using Completion Ports":
"...the data gets copied by AFD.SYS to its internal buffers (up to the
SO_SNDBUF setting)..." (see "Who Manages the Buffers?").
http://msdn.microsoft.com/msdnmag/issues/1000/winsock/default.aspx

As I see it, send() function must not return WSAENOBUFS error
regardless of the socket is blocking or nonblocking. See also MSDN
KB201213 "BUG: Send() Fails with Error WSAENOBUFS Over Blocking
Socket". On nonblocking socket can be returned WSAEWOULDBLOCK if buffer
space is unavailable or returned number of sent bytes between 1 and the
requested length. Also other errors like WSAENETDOWN can be returned
but not WSAENOBUFS error. So current WinSock API behavior is a BUG.

Also, in my opinion, current nonblocking socket behavior can be
unexpected and Microsoft should document it better and more
unambiguously.

[Arkady Frenkel]

Sure, but if you want that variant you can do it on different thread
Arkady

"DSheva" <dsh...@gmail.com> wrote in message

news:1125567515.3...@g14g2000cwa.googlegroups.com...

[Alexander Nickolov]

I can't argue with your sentiments, I feel much the same here :).
However, you are giving here a link to an article, not the
WinSock documentation. Don't confuse magazines with
documentation.

--
=====================================
Alexander Nickolov
Microsoft MVP [VC], MCSD
email: agnic...@mvps.org
MVP VC FAQ: http://www.mvps.org/vcfaq
=====================================

"DSheva" <dsh...@gmail.com> wrote in message

news:1125570956....@g44g2000cwa.googlegroups.com...

[qfel]

> 1a. Succeeded and all data is processed to sending when data size is
> more then socket send buffer size (SO_SNDBUF), for example - send 64Mb
> [Not documented, data must be partially processed].

How do you test it?
If your sending data to local host I guess it is fine - despite buffer size
send is able to deliver all data immediately (I think I saw written
somewhere that 'local' sockets are implemented specifically to achieve
better performance.

[DSheva]

Hello.

I have tested it on two hosts (on next test pairs: 1. Windows XP SP2 +
Windows 2003 Server; 2. Windows XP SP2 + Windows XP SP1; 3. Windows XP
SP1 + Windows 2003 Server; 4. Windows XP SP2 + Windows 2000 SP4) on LAN
(100Mbps Fast Ethernet).

As I see it, send() function behavior must not depend on localhost or
not localhost is used. It must works as stated in documentation in any
case.

Best regards,
DS

[DSheva]

Alexander, hello!

Yes, but article written by Microsoft developers (Microsoft Windows
2000 Networking group) and published by Microsoft. ;)
Alexander, thank you for your help. :)

[DSheva]

Hello.

Some additional and must be most believable information from MSDN about
SO_SNDBUF and send() behaviour.

See article "INFO: Design Issues - Sending Small Data Segments Over TCP
w/Winsock" in MSDN Knowledge Base:

"If necessary, Winsock can buffer significantly more than the SO_SNDBUF
buffer size."

"Winsock uses the following rules to indicate a send completion to the
application (depending on how the send is invoked, the completion
notification could be the function returning from a blocking call,
signaling an event or calling a notification function, and so forth):

- If the socket is still within SO_SNDBUF quota, Winsock copies the
data from the application send and indicates the send completion to the
application.

- If the socket is beyond SO_SNDBUF quota and there is only one
previously buffered send still in the stack kernel buffer, Winsock
copies the data from the application send and indicates the send
completion to the application.

- If the socket is beyond SO_SNDBUF quota and there is more than one
previously buffered send in the stack kernel buffer, Winsock copies the
data from the application send. Winsock does not indicate the send
completion to the application until the stack completes enough sends to
put the socket back within SO_SNDBUF quota or only one outstanding send
condition."

Best regards,
DS