WSASend() - Multithreading
Bob Gaffaney
current form, each connected socket has a single read
thread while any number of clients can call WSASend.
I find that in many cases, even though WSASend succeeds
and reports all of the bytes sent, the message never makes
it to the wire, as confirmed by an Ethernet sniffer.
It seems that sharing the socket among several clients is
the likely cause of this. I tried serializing access to
the socket with a critical section, but this does not fix
this problem.
I have tried both blocking and overlapped calls and this
makes no difference.
Anyone have experience with this? Thanks in advance for
any light you can provide on this.
Thanks,
Bob Gaffaney
Siemens SBT
Bob Gaffaney
If you post another send() on the same connection while a
first send() is still in the works, winsock may chose to
append some or all of the data from the second operation.
e.g. If two threads each send 1000 bytes, the application
on the far end may recv() two packets of 1000 bytes; one
packet of 2000 bytes or even one of 1440 bytes and one of
560 bytes.
Logically, what I am sending are datagrams - though end to
end delivery integrity is required so we are using TCP not
UDP.
If there is a way to make Winsock to send the messages as
sent that would be preferable, else I will have to send
them in a single thread and provide enough information to
disassemble them on the far end.
Any suggestions?
Thanks
Bob Gaffaney
>.
>
arkadyf
From your second message I can understand that all bytes were received on
other side , simple , not in the sequence you want . Is it true ?
In this case it's OK . Because tcpip.sys/afd.sys ( kernel part of
winsock ) divide the data as MTU stand ( for ethernet it's 1500 , where 1460
bytes is data and 40- IP+TCP headers ).
If Nagle algorithm On ( usually ) the packet not send up to ACK on previous
send not received and during that period data
gathered to be send . You can switch it off to send data immediately but in
the case of short packets you'll make the network life hard ( like you send
1 byte of data with 40 bytes of IP&TCP header which mean overhead of
4000% ). Because of that usually it's not recommended , and better to set
the length at the beginning of message . At this case you can send packets
from different threads but at the remote host you have "enough information
to disassemble them on the far" ( as you wrote )
TCP/IP is a stream protocol so you always receive all your data ( and in the
sequence you send ) , but no guarantee that on
receiving site you'll receive the same packets in length as you send.
So when WSASend() return number of bytes , it says the data in system buffer
( not yet sent ).
OTTO if you don't use some kind of serialization or
blocking sockets , no guarantee that data will not be mix ( on the sending
side )because using the same socket from different threads.
Arkady
"Bob Gaffaney" <bob.gaf...@sbt.siemens.com> wrote in message
news:1695d01c227bc$e032b670$9ae62ecf@tkmsftngxa02...
Len Holgate
> end delivery integrity is required so we are using TCP not
> UDP.
Bob,
TCP is a byte stream. The developer always has to do the work required to
break the byte stream into protocol specific chunks. From what you describe,
the easiest way to do this would be to include a header in your 'datagram'
and have that header contain the length of the packet. Then you read the
header, work out how many more bytes make up one packet and process that
packet.
We have some articles on our website which cover this kind of thing:
http://www.jetbyte.com/portfolio-showarticle.asp?articleId=37&catId=1&subcat
Id=2
--
Len Holgate
http://www.jetbyte.com
The right code, right now.
Contract Programming and Consulting Services.
Alun Jones
>If you post another send() on the same connection while a
>first send() is still in the works, winsock may chose to
>append some or all of the data from the second operation.
This is TCP's choice, not Winsock's. All TCP stacks do this.
>Logically, what I am sending are datagrams - though end to
>end delivery integrity is required so we are using TCP not
>UDP.
>
>If there is a way to make Winsock to send the messages as
>sent that would be preferable, else I will have to send
>them in a single thread and provide enough information to
>disassemble them on the far end.
That's exactly what you'll have to do. Fine, you've decided that TCP's
reliability is needed, so you use TCP; but TCP doesn't care about your
datagrams, and does nothing to indicate where they start or end - you'll have
to do that. You'll have to insert bytes into the stream that your application
can use to determine where the "datagrams" begin and end, and you'll have to
ensure that you've read every byte of each one before you process it.
The most common methods to use are delimiter characters (or character
sequences), with quoting to allow the delimiter to appear in the message
itself, or length counts.
Alun.
~~~~
[Please don't email posters, if a Usenet response is appropriate.]
--
Texas Imperial Software | Try WFTPD, the Windows FTP Server. Find us at
1602 Harvest Moon Place | http://www.wftpd.com or email al...@texis.com
Cedar Park TX 78613-1419 | VISA/MC accepted. NT-based sites, be sure to
Fax/Voice +1(512)258-9858 | read details of WFTPD Pro for NT.
Thomas F. Divine
own "message oriented" private protocol in your TCP stream.
This would involve adding a fixed-size header (more then just a "delimiter")
to each "message" that is sent by any thread. The fixed header would
identify the contents of each message and possibly have a sequence number,
etc. Most importantly, the fixed header would include a length field that
identifies the length of data in your "message". The receiver would always
read for the fixed header. Once found, examine header. From header length
field, make as many reads as necessary to read the length and then go
process that.
Your receive will have to buffer received data and glue it together based on
header information. Then pass received "message" for processing.
May sound like a little more work then you want to do. However, if you do it
systematically it has many virtues.
Good luck,
--
Thomas F. Divine
PCAUSA - Tools & Resources For Network Software Developers
NDIS Protocol/Intermediate/Hooking - TDI Client/Filter
<http://www.pcausa.com> - <http://www.rawether.net>
"Alun Jones" <al...@texis.com> wrote in message
news:dqWW8.200$p72.17...@newssvr11.news.prodigy.com...
John Duddy
will be intermixed like this:
WSASend 1: buffer = a b c d
WSASend 2: buffer = 1 2 3 4
Received: a b 1 2 3 c d 4
?
OR are you saying there's no guarantee in which order the blobs "1234" and
"abcd" will be received?
JD
"arkadyf" <ark...@hotmail.com> wrote in message
news:uOKTMX9JCHA.1608@tkmsftngp09...
[snip]
Bob Gaffaney
TCP's discretion - i.e. may get 'a' and half of 'b' in one
received message at the other end, and then the other half
of 'b' and '1' in the next.
Thanks,
Bob
>.
>
John Duddy
was one, four-byte buffer sent in a single WSASend() operation, and "1234"
was another, 4-byte buffer also sent in a single WSASend operation?
JD
"Bob Gaffaney" <bob.gaf...@sbt.siemens.com> wrote in message
news:149e401c22841$a0f48580$a5e62ecf@tkmsftngxa07...
arkadyf
That's exactly the question that unclear for me. If Bob use serialization
or blocked sockets it will be OK , but if nonblock/async writing of one
thread interrupted by other the same
nonblock/async writing on the same socket how afd.sys operate in
this case ? That's why I advice to use serialization or blocked sockets (
To work for sure :) ) . The question is if afd.sys support
different copying buffers for each thread for the same socket ?
Arkady
"John Duddy" <jdu...@idontwantyourunsolicitedmsgDOTstbernard.com> wrote in
message news:uip544c...@corp.supernews.com...
John Duddy
guarantee that the buffers would not be intermixed. That is a strong
statement, and in my mind directly contradicts this statement from MSDN:
Returning from this function allows invocation of another pending completion
routine for this socket. All waiting completion routines are called before
the alertable thread's wait is satisfied with a return code of
WSA_IO_COMPLETION. The completion routines can be called in any order, not
necessarily in the same order the overlapped operations are completed.
However, the posted buffers are guaranteed to be sent in the same order they
are supplied.
So, have you seen this intermingling occur? I'd hate to impose unnecessary
serialization on a high-performance app unless it is necessary, and then
only as much as is needed to ensure safe, predictable operation.
Thanks -
JD
"arkadyf" <ark...@hotmail.com> wrote in message
news:eA5JbbLKCHA.2320@tkmsftngp11...
Mike Liu (MS)
Once data is honored to be sent, the stack will guarantee the order of the
stream as accepted; otherwise you would see data corruption.
On the other hand, some kind of synchronization has to be in place. For
example, if you request "abcd" to be sent, it is possible only part of the
request will be honored and only "ab" will be sent, If the other send
request of "1234" is fully honored, the receiving end will see "ab1234".
Thus, it is critical to verify how many bytes are actually sent for each
operation and implement the synchronization mechanism accordingly.
Thanks,
Mike Liu
Microsoft Developer Support
This posting is provided "AS IS" with no warranties, and confers no rights.
John Duddy
the buffer has been sent, requiring another I/O call? Is there a way to tell
it "send this buffer in its entirety and contiguously, and tell me when you
have finished (or had a catastrophic failure)"? Doesn't WriteFile behave
this way?
JD
"Mike Liu (MS)" <mike...@online.microsoft.com> wrote in message
news:Na2X10fKCHA.2328@cpmsftngxa08...
arkadyf
I said this , because once was knocked by such problem and quickly put
critical section ( which much quicker then other sync methods but
good only for threads of the same process ). So my thought was that it's
because multithreading app behavior , but , as Mike cleared now , it's
because Afd.sys not guarantee that all the buffer taken at once from
user to kernel ( ordinary situation which can happen in one thread app , but
there it's not the problem because only one buffer ).
Thanks for clarification , Mike
Thank John for pointed on it , I don't like unneeded synchronization too (
but how many times Sleep(0) saved the situation :) ! ) so I wanted to check
that situation .
Arkady
"John Duddy" <jdu...@idontwantyourunsolicitedmsgDOTstbernard.com> wrote in
message news:uiuvc5i...@corp.supernews.com...
Alun Jones
wrote:
>I said this , because once was knocked by such problem and quickly put
>critical section ( which much quicker then other sync methods but
>good only for threads of the same process ). So my thought was that it's
>because multithreading app behavior , but , as Mike cleared now , it's
>because Afd.sys not guarantee that all the buffer taken at once from
>user to kernel ( ordinary situation which can happen in one thread app , but
>there it's not the problem because only one buffer ).
It's worth noting that a common - and effective - solution to this is to have
each socket only writable by one thread. That way, you can not ever
intermingle two sets of data. A mutex on the socket or its buffer is
generally a lot of trouble, and can lead to messy code - besides, the socket
is a serial device, and therefore isn't going to benefit from parallelisation,
until and unless the following becomes true:
1. You're running on multiple processors.
2. Your immediate processing on network input takes longer than the traffic
takes to come in.
For many - perhaps even most - applications, the second part is false.
John Duddy
multiple WSASend operations. If you get two overlapped WSASends started, one
after the other, is there any reason to assume that they will not be sent
sequentially and in theiur entirety?
The only time multiple threads can come into play in this scenario is when
you are using an IOCP with multiple threads servicing it. If sequntially
issued overlapped WSASend operations do not complete sequentially and in
their entirety (i.e. partial completion requiring re-issuing the WSASend for
the unsent portion of the buffer) then IOCP is essentially crippled in
winsock applications. I can see no other conclusion.
All of this was started by someone saying the buffers get intermingled. If
that is so, then you can never safely use IOCP with winsock. Can anyone else
see another conclusion, or should we re-examine the assertion that the
buffers sent via two overlapped WSASend operations can be intermingled?
Thanks for your patience for my pedanticism ;-}
JD
writes.
"Alun Jones" <al...@texis.com> wrote in message
news:s6hY8.5751$jJ2.2028823720@newssvr12.news.prodigy.com...
Len Holgate
We've been doing a lot of work in this area recently. We have a TCP/UDP test
harness that we have been testing our IOCP servers with and it exposed a
problem with multiple WSASend() operations being executed out of sequence.
However, this was merely because our design uses the IOCP to "marshal" the
socket IO calls into the IO thread pool rather than executing the sends in
user threads that may, potentially, be terminated before the IO completes.
Because we're testing on a multi-processor box and because we have multiple
IO threads servicing our IOCP the WSASend() requests were suffering from the
problem more usually associated with multiple WSARecv()s and our test
harness flushed that out. The problem was that the actual calls to WSASend()
were being processed out of sequence. After putting in some code to ensure
that the buffers for each send had a sequence number in them that was
incremented by each write operation, and some code in the IOCP worker thread
to ensure that the send operations were actually executed in the required
sequence our problems went away.
I have never seen an overlapped WSASend() complete and report a partial
buffer transmission. I cannot believe that this could occur except in the
case where the connection is closed mid send and even then I've never seen
it. As you point out, if the WSASend() could complete and indicate that it
had only sent part of the buffer then the whole IOCP model would be fatally
flawed because there's no way of knowing how many other send's are queued
up, so there's no way of ever being able to recover and send the rest of the
partially transmitted buffed in the correct place in the byte stream.
I have never seen buffers intermingled in the way that is suggested by the
other poster either. As you say, that would also make the IOCP model
unworkable.
My tests have made me confident that these problems with the API just do not
exist. There are plenty of programmer induced problems that could cause
similar behavior, but they're just bugs ;)
In case you're interested, we have an article on our web site about the how
to have multiple outstanding WSARecv() and WSASend() operations operate as
expected in a multi threaded, multi processor, IOCP design.
http://www.jetbyte.com/portfolio-showarticle.asp?articleId=44&catId=1&subcat
Id=2
And another article about our C# server test tool. This tool provides a
framework that you can plug protocol specific tests into, and makes sure
that the server can deal with message fragmentation, partial and multiple
messages, etc.
http://www.jetbyte.com/portfolio-showarticle.asp?articleId=46&catId=4&subcat
Id=11
Each article comes with complete source code.
John Duddy
building would need some chewing gum and bailing wire to keep it together.
JD
"Len Holgate" <Len.H...@JetByte.com> wrote in message
news:3d3311cd$0$8509$cc9e...@news.dial.pipex.com...
arkadyf
return so number of bytes as send ) is when you have not enough
nonpageable memory for buffer afd.sys try to allocate to move you buffer
into this one and not send fail which will happen later and driver return
the answer about it.
Happen very rear but ...gum and wire , I'm not so sure.
Arkady
"John Duddy" <jdu...@idontwantyourunsolicitedmsgDOTstbernard.com> wrote in
message news:uj66hnt...@corp.supernews.com...
Len Holgate
> The first example when you do receive WSASend not succeed ( and cannot
> return so number of bytes as send ) is when you have not enough
> nonpageable memory for buffer afd.sys try to allocate to move you buffer
> into this one and not send fail which will happen later and driver return
> the answer about it.
> Happen very rear but ...gum and wire , I'm not so sure.
Would this occur at the point where you make the WSASend call, rather than
via the completion port? If it occurs at that point you could, maybe, still
be able to do someting about it. Either way, if you're out of non paged pool
you're probably screwed anyway...
arkadyf
memory back to pool .
BTW for me ( and I don't know why ) it happen on non-blocked BCD sockets
from few threads without IOCP . It happen once and I don't know
really why , maybe I was drunk or computer hate me that day :)
Arkady
"Len Holgate" <Len.H...@JetByte.com> wrote in message
news:3d36d3e2$0$238$cc9e...@news.dial.pipex.com...