How to close a threaded, blocking accept and recv
Trouble@Mill
After the very helpful answers to any earlier post, I now have my
'threaded' server running pretty well. But, I want to make sure that,
when it comes time to shut the beast down, it does so as cleanly as
possible.
First, a little background. The server is designed to control a
network device that does most of it's 'chattering' over UDP,
incessantly. It also needs to support a (very small) number of
clients that connect over TCP to pass commands to the device. In
fact, the device itself is also a client.
The main thread is used for the UDP communication, in both directions,
to the device. Before dropping into the 'main' processing loop, it
spawns off a thread that handles the TCP accept. For each connection,
this thread then spawns off another 'worker' thread that accepts the
input and passes the 'commands' back to the main loop. Both the
accept and recv in these threads are 'blocking' calls.
Now, when it comes time to shut down, I can close all the sockets in
the 'main' thread. This causes the accept (thread) to fail with an
'Interrupted Function Call (10004)' error. So, do I treat this as the
equivalent of 'shutdown now' for the thread. Or is there some other
way to gracefully shut down this thread.
Plus, the 'worker' threads are just terminated because the 'main'
thread has ended. Again, is this an acceptable way of handling
things. Or should I be trying to 'gracefully' terminate the threads.
What other ways are there for me to handle this kind of scenario.
Currently, this is running on Windows, because it's easier to debug it
in VC, but eventually it will run predominantly under Linux. So far,
the only difference I've seen between the platforms is, under Linux,
the 'accept thread' doesn't get the 10004 error. It just gets killed
because the main thread has gone.
Cheers,
Eddie
David Schwartz
"Trouble@Mill" <Trouble@Mill.> wrote in message
news:3706pvokcq7f8jbna...@4ax.com...
> Now, when it comes time to shut down, I can close all the sockets in
> the 'main' thread. This causes the accept (thread) to fail with an
> 'Interrupted Function Call (10004)' error. So, do I treat this as the
> equivalent of 'shutdown now' for the thread. Or is there some other
> way to gracefully shut down this thread.
This is dangerous, though it's generally considered acceptable if it's
used only for a shutdown of the entire process. More elegant is to set a
'shutdown now' flag and then just make a connection, causing the 'accept' to
return.
> Plus, the 'worker' threads are just terminated because the 'main'
> thread has ended. Again, is this an acceptable way of handling
> things. Or should I be trying to 'gracefully' terminate the threads.
It is acceptable but not graceful. For active TCP connections, you can
use 'shutdown'. For UDP sockets, you can send a dummy datagram after setting
a flag. Closing a socket while it's in use is not considered bad practice
because race conditions are unavoidable, however these are generally
considered to be an acceptable risk if and only if they are used only when
the entire process is being shut down.
> What other ways are there for me to handle this kind of scenario.
Don't use blocking socket calls.
> Currently, this is running on Windows, because it's easier to debug it
> in VC, but eventually it will run predominantly under Linux. So far,
> the only difference I've seen between the platforms is, under Linux,
> the 'accept thread' doesn't get the 10004 error. It just gets killed
> because the main thread has gone.
You don't really have to shut all your threads down cleanly. However,
it's a good idea to be able to do so, especially if your code is a library.
DS
Sami Korhonen
> the 'main' thread. This causes the accept (thread) to fail with an
> 'Interrupted Function Call (10004)' error. So, do I treat this as the
> equivalent of 'shutdown now' for the thread. Or is there some other
> way to gracefully shut down this thread.
>
closesocket() is the only way to cancel blocking/pending recv/send/accept
from another thread. Another solution would be creating global event object
and using GetOverlappedResult (this way you can implement custom timeouts):
INT Send(SOCKET Socket, LPVOID lpBuffer, DWORD dwBuffer, LPWSAOVERLAPPED
lpOverlapped)
{
WSABUFFER Buffer;
HANDLE hEvents[2];
DWORD dwBytesSent, dwFlags;
Buffer.buf = lpBuffer;
Buffer.len = dwBuffer;
hEvents[0] = hShutDown;
hEvents[1] = lpOverlapped->hEvent;
if (! WSASend(Socket, &Buffer, 1, &dwBytesSent, 0, lpOverlapped, 0))
{
if (WSAGetLastError() != ERROR_IO_PENDING) return -1;
switch (WaitForMultipleObjects(2, hEvents, FALSE, 30000))
{
case WAIT_OBJECT_0:
return -2;
case WAIT_OBJECT_1:
if (GetOverlappedResult(Socket, lpOverlapped, &dwBytesSent,
FALSE, &dwFlags))
{
return (INT)dwBytesSent;
}
else return -1;
default:
WSASetLastError(WSAETIMEDOUT);
return -1;
}
}
else if (WaitForSingleObject(hShutDown, 0) == WAIT_OBJECT_0) return -2;
return (INT)dwBytesSent;
}
INT Recv(SOCKET Socket, LPVOID lpBuffer, DWORD dwBuffer, LPWSAOVERLAPPED
lpOverlapped)
{
WSABUFFER Buffer;
HANDLE hEvents[2];
DWORD dwBytesReceived, dwFlags;
Buffer.buf = lpBuffer;
Buffer.len = dwBuffer;
hEvents[0] = hShutDown;
hEvents[1] = lpOverlapped->hEvent;
dwFlags = 0;
if (! WSARecv(Socket, &Buffer, 1, &dwBytesReceived, &dwFlags,
lpOverlapped, 0))
{
if (WSAGetLastError() != ERROR_IO_PENDING) return -1;
switch (WaitForMultipleObjects(2, hEvents, FALSE, 30000))
{
case WAIT_OBJECT_0:
return -2;
case WAIT_OBJECT_1:
if (GetOverlappedResult(Socket, lpOverlapped, &dwBytesReceived,
FALSE, &dwFlags))
{
return (INT)dwBytesReceived;
}
else return -1;
default:
WSASetLastError(WSAETIMEDOUT);
return -1;
}
}
else if (WaitForSingleObject(hShutDown, 0) == WAIT_OBJECT_0) return -2;
return (INT)dwBytesReceived;
}
On success & socket error, functions behave exactly like send()/recv(). On
server shutdown they both return -2.
Tom Smith
> <code>
>
> On success & socket error, functions behave exactly like send()/recv(). On
> server shutdown they both return -2.
>
Would work, but perhaps the OP should ditch blocking calls altogether :)
Trouble@Mill
<dav...@webmaster.com> wrote:
David, firstly, thank you for your answers. Not just to my questions,
but your contribution to the group as a whole.
>"Trouble@Mill" <Trouble@Mill.> wrote in message
>news:3706pvokcq7f8jbna...@4ax.com...
>
>> Now, when it comes time to shut down, I can close all the sockets in
>> the 'main' thread. This causes the accept (thread) to fail with an
>> 'Interrupted Function Call (10004)' error. So, do I treat this as the
>> equivalent of 'shutdown now' for the thread. Or is there some other
>> way to gracefully shut down this thread.
>
> This is dangerous, though it's generally considered acceptable if it's
>used only for a shutdown of the entire process. More elegant is to set a
>'shutdown now' flag and then just make a connection, causing the 'accept' to
>return.
OK. I already have a 'global' shutdown flag, so to add a socket(),
connect(), and closesocket() 'aint going to be a problem. That takes
care of the accept() thread.
>> Plus, the 'worker' threads are just terminated because the 'main'
>> thread has ended. Again, is this an acceptable way of handling
>> things. Or should I be trying to 'gracefully' terminate the threads.
>
> It is acceptable but not graceful. For active TCP connections, you can
>use 'shutdown'. For UDP sockets, you can send a dummy datagram after setting
>a flag. Closing a socket while it's in use is not considered bad practice
>because race conditions are unavoidable, however these are generally
>considered to be an acceptable risk if and only if they are used only when
>the entire process is being shut down.
In fact, the entire process *is* being shut down at this point. The
UDP side is already being taken care of. So I guess what you're
implying is that whenever I spawn off a new TCP thread, I store the
socket in a 'global' list, so that the main thread can then chase down
this list, doing a closesocket() against each. I'm guessing that will
cause each of the threads to fail with a 'known' error that I can
check for. Much like the way the thread checks for the client closing
the connection and terminating.
>> What other ways are there for me to handle this kind of scenario.
>
> Don't use blocking socket calls.
I though of that. But doesn't that put me in the realm of polling
loops, which could end up with a higher processing 'overhead'. Plus,
the thread couldn't always react in a timely manner, unless it was a
very small interval between polls.
>> Currently, this is running on Windows, because it's easier to debug it
>> in VC, but eventually it will run predominantly under Linux. So far,
>> the only difference I've seen between the platforms is, under Linux,
>> the 'accept thread' doesn't get the 10004 error. It just gets killed
>> because the main thread has gone.
>
> You don't really have to shut all your threads down cleanly. However,
>it's a good idea to be able to do so, especially if your code is a library.
But I'm such a neat and tidy person :-) Although, I'm sure my wife
would disagree.
Cheers,
Eddie
David Schwartz
"Trouble@Mill" <Trouble@Mill.> wrote in message
> On Sun, 19 Oct 2003 16:58:12 -0700, "David Schwartz"
> <dav...@webmaster.com> wrote:
> David, firstly, thank you for your answers. Not just to my questions,
> but your contribution to the group as a whole.
You are welcome.
> OK. I already have a 'global' shutdown flag, so to add a socket(),
> connect(), and closesocket() 'aint going to be a problem. That takes
> care of the accept() thread.
Let the thread that was using the socket do the 'closesocket' call. That
way you can be 100% sure it won't call 'accept' after the socket is closed.
That's maximally clean. However, again, since you're just shutting down the
whole process anyway, you don't have to be perfectly clean unless you want
to.
> > It is acceptable but not graceful. For active TCP connections, you
can
> >use 'shutdown'. For UDP sockets, you can send a dummy datagram after
setting
> >a flag. Closing a socket while it's in use is not considered bad practice
> >because race conditions are unavoidable, however these are generally
> >considered to be an acceptable risk if and only if they are used only
when
> >the entire process is being shut down.
> In fact, the entire process *is* being shut down at this point. The
> UDP side is already being taken care of. So I guess what you're
> implying is that whenever I spawn off a new TCP thread, I store the
> socket in a 'global' list, so that the main thread can then chase down
> this list, doing a closesocket() against each. I'm guessing that will
> cause each of the threads to fail with a 'known' error that I can
> check for. Much like the way the thread checks for the client closing
> the connection and terminating.
No, call 'shutdown' against each TCP connection. That will cause the
threads to fail with an error you can check for, and then those threads can
call 'closesocket'. The scenario you're trying to avoid is where the
shutdown thread calls 'closesocket' just before the TCP thread calls, say,
'recv'.
> >> What other ways are there for me to handle this kind of scenario.
> >
> > Don't use blocking socket calls.
> I though of that. But doesn't that put me in the realm of polling
> loops, which could end up with a higher processing 'overhead'. Plus,
> the thread couldn't always react in a timely manner, unless it was a
> very small interval between polls.
No, not at all. Winsock has many efficient forms of non-blocking I/O.
Take a look at completion events, completion routines, I/O completion ports,
and various other techniques.
> > You don't really have to shut all your threads down cleanly. However,
> >it's a good idea to be able to do so, especially if your code is a
library.
> But I'm such a neat and tidy person :-) Although, I'm sure my wife
> would disagree.
:)
DS
Trouble@Mill
<dav...@webmaster.com> wrote:
>
>"Trouble@Mill" <Trouble@Mill.> wrote in message
>news:h6t7pvsvnoiotfs9t...@4ax.com...
>
>> On Sun, 19 Oct 2003 16:58:12 -0700, "David Schwartz"
>> <dav...@webmaster.com> wrote:
>
>> David, firstly, thank you for your answers. Not just to my questions,
>> but your contribution to the group as a whole.
>
> You are welcome.
>
>> OK. I already have a 'global' shutdown flag, so to add a socket(),
>> connect(), and closesocket() 'aint going to be a problem. That takes
>> care of the accept() thread.
>
> Let the thread that was using the socket do the 'closesocket' call. That
>way you can be 100% sure it won't call 'accept' after the socket is closed.
>That's maximally clean. However, again, since you're just shutting down the
>whole process anyway, you don't have to be perfectly clean unless you want
>to.
That's what I was planning to do. The closesocket() I referred to was
for the socket that I used to make a connection() which then causes
the accept to wake up and then gracefully terminate.
>> > It is acceptable but not graceful. For active TCP connections, you
>can
>> >use 'shutdown'. For UDP sockets, you can send a dummy datagram after
>setting
>> >a flag. Closing a socket while it's in use is not considered bad practice
>> >because race conditions are unavoidable, however these are generally
>> >considered to be an acceptable risk if and only if they are used only
>when
>> >the entire process is being shut down.
>
>> In fact, the entire process *is* being shut down at this point. The
>> UDP side is already being taken care of. So I guess what you're
>> implying is that whenever I spawn off a new TCP thread, I store the
>> socket in a 'global' list, so that the main thread can then chase down
>> this list, doing a closesocket() against each. I'm guessing that will
>> cause each of the threads to fail with a 'known' error that I can
>> check for. Much like the way the thread checks for the client closing
>> the connection and terminating.
>
> No, call 'shutdown' against each TCP connection. That will cause the
>threads to fail with an error you can check for, and then those threads can
>call 'closesocket'. The scenario you're trying to avoid is where the
>shutdown thread calls 'closesocket' just before the TCP thread calls, say,
>'recv'.
My bad. I meant to say shutdown(), but it came out as closesocket().
>> >> What other ways are there for me to handle this kind of scenario.
>> >
>> > Don't use blocking socket calls.
>
>> I though of that. But doesn't that put me in the realm of polling
>> loops, which could end up with a higher processing 'overhead'. Plus,
>> the thread couldn't always react in a timely manner, unless it was a
>> very small interval between polls.
>
> No, not at all. Winsock has many efficient forms of non-blocking I/O.
>Take a look at completion events, completion routines, I/O completion ports,
>and various other techniques.
Can you suggest somewhere I might e able to read up on these.
How many of these techniques are also available to the socket
implementations on Linux.
David Schwartz
"Trouble@Mill" <Trouble@Mill.> wrote in message
> > No, not at all. Winsock has many efficient forms of non-blocking I/O.
> >Take a look at completion events, completion routines, I/O completion
ports,
> >and various other techniques.
> Can you suggest somewhere I might e able to read up on these.
Google for 'winsock' 'completion' 'overlapped' 'event'. You should be
able to find many articles.
> How many of these techniques are also available to the socket
> implementations on Linux.
None of the good ones are portable. How you do efficient I/O on Windows
is just different from how you do it on UNIX. You can use 'select' on both,
but it's optimal on neither.
DS
Sami Korhonen
>
> > Can you suggest somewhere I might e able to read up on these.
>
> Google for 'winsock' 'completion' 'overlapped' 'event'. You should be
> able to find many articles.
>
> > How many of these techniques are also available to the socket
> > implementations on Linux.
>
> None of the good ones are portable. How you do efficient I/O on
Windows
> is just different from how you do it on UNIX. You can use 'select' on
both,
> but it's optimal on neither.
If you decide to use some other solution than select() on windows, you
should create io and worker thread pool(s). Advantage of this type of
solution is, that only io routines need to be written twice (which usually
isn't much). Here's somewhat simplified version of such solution using iocp
on windows, and select on bsd-socket compatible system:
#ifdef _WIN32
int IoThread(void)
{
for (;;)
{
GetQueuedCompletionStatus();
// Get context from overlapped structure
switch (lpContext->dwID)
{
case READ_SOCKET:
// Send to worker thread if enough data has been received
QueueJob();
// Otherwise receive more
WSARecv();
break;
case WRITE_SOCKET:
// Send to worker thread if enough data has been sent
QueueJob();
// Otherwise send more
WSASend();
break;
case READ_FILE:
// ...
ReadFile();
break;
case WRITE_FILE:
// ...
WriteFile();
break;
}
}
}
#else
int IoThread(void)
{
for (;;)
{
select();
// Find signaled filedescriptor
switch (lpContext->dwID)
{
case READ_SOCKET:
recv();
// Send to worker thread, when enough data has been received
QueueJob();
break;
case WRITE_SOCKET:
send();
// Send to worker thread, when enough data has been sent
QueueJob();
break;
case READ_FILE:
read();
// ...
break;
case WRITE_FILE:
write();
// ...
break;
}
}
}
#endif
Instead of select() you could also use
http://www.monkey.org/~provos/libevent/ library for unix.
SenderX
> threads to fail with an error you can check for, and then those threads
can
> call 'closesocket'.
Are you sure shutdown will cancel pending reads?
I don't think it does.
SenderX
That will do it.
;)
--
The designer of the experimental, SMP and HyperThread friendly, AppCore
library.
http://AppCore.home.comcast.net
Trouble@Mill
> No, call 'shutdown' against each TCP connection. That will cause the
>threads to fail with an error you can check for, and then those threads can
>call 'closesocket'. The scenario you're trying to avoid is where the
>shutdown thread calls 'closesocket' just before the TCP thread calls, say,
>'recv'.
David,
That didn't work, well not on Windows :( Even after the shutdown()
using the same socket handle as the recv(), the recv() continued to
wait. It didn't end. Any thoughts ??
However, running the *exact* same code on Linux, the recv() ended with
zero bytes received when the shutdown() was issued. Which I use as an
indication to the thread to close gracefully.
Cheers,
Eddie
Sami Korhonen
"SenderX" <x...@xxx.xxx> wrote in message
news:z%0lb.321913$mp.2...@rwcrnsc51.ops.asp.att.net...
David Schwartz
"Trouble@Mill" <Trouble@Mill.> wrote in message
> > No, call 'shutdown' against each TCP connection. That will cause the
> >threads to fail with an error you can check for, and then those threads
can
> >call 'closesocket'. The scenario you're trying to avoid is where the
> >shutdown thread calls 'closesocket' just before the TCP thread calls,
say,
> >'recv'.
> David,
> That didn't work, well not on Windows :( Even after the shutdown()
> using the same socket handle as the recv(), the recv() continued to
> wait. It didn't end. Any thoughts ??
Really?! You did a shutdown(SD_BOTH)?
> However, running the *exact* same code on Linux, the recv() ended with
> zero bytes received when the shutdown() was issued. Which I use as an
> indication to the thread to close gracefully.
Yeah, that's what should have happened. If that doesn't work correctly
under Windows, then a graceful shutdown from another thread without
coordination may not be possible.
DS
David Schwartz
"Sami Korhonen" <darkone@ioftpd_KILLTHESPAM.com> wrote in message
> Yep, it doesn't do that on windows.
Then I think you'll have to coordinate the two threads. :(
I don't know of any other way.
DS