How to wait on completion port and event?

[Andy Johnson]

Hi,

I'm modifying some server code that uses overlapped socket I/O and
completion ports. It currently calls GetQueuedCompletionStatus() to
block until data arrives. I need to change this so that it blocks
until data arrives OR a win32 event (not a wsa event) is signalled.
I'm hoping someone can help me understand how to do this.

More detail:

I understand how to do this for a normal socket:

SOCKET mySocket = socket( ... );
HANDLE hMyEvent = CreateEvent( ... );
...
HANDLE hNetworkEvent = WSACreateEvent();
WSAEventSelect(mySocket, hNetworkEvent, FD_READ);
HANDLE eventHandles[] = { m_hMyEvent, hNetworkEvent };
DWORD nEvent = WaitForMultipleObjects(2, eventHandles, FALSE,
INFINITE);

I'd like to achieve something similar for the completion port. Clearly
the above won't work because WSAEventSelect needs a SOCKET, not a
completion port handle. I know I could poll the completion port and
event in turn but that has obvious problems.

This seems like something that would often be needed, but I can't see
how to do it. I'd appreciate any help on this.

Thanks,

Andy

[David Schwartz]

On Sep 3, 1:55 am, Andy Johnson <andyjohns...@gmail.com> wrote:

> I'm modifying some server code that uses overlapped socket I/O and
> completion ports. It currently calls GetQueuedCompletionStatus() to
> block until data arrives. I need to change this so that it blocks
> until data arrives OR a win32 event (not a wsa event) is signalled.
> I'm hoping someone can help me understand how to do this.

Create another thread to block on the IOCP stuff. Have this thread
block on the Win32 event. If needed, you can queue a packet to the
completion port (with 'PostQueuedCompletionStatus') when you receive
an event to get the other thread's attention.

One thread should not be doing two fundamentally different things at
the same exact time.

DS

[Andy Johnson]

David,

Thanks for responding.

The event is used to tell the thread to exit, so it is logically part
of the thread's 'purpose'. However, I take your point.

If I understand it, your solution is that a second thread (b) waits on
the event and wakes up the thread waiting on the completion port (a)
by using PostQueuedCompletionStatus() to inject a completion packet.
Presumably I would use a special value for the completion key to
indicate that the event has signaled and that thread a should
terminate. Is this correct?

I like this solution. The problem I have is that there is one 'exit'
event, but many instances of thread a reading from the completion
port. The thread b would therefore have to ensure that each reading
thread exited properly (by sending multiple completion packets?).
That, and the extra thread to watch the event, adds up to a lot of
overhead. Intuitively, I feel that this should be simpler - but that
may be my lack of experience.

I feel that what I'm doing is probably quite common. Is this a
standard implementation pattern for this problem?

Andy

[David Schwartz]

On Sep 3, 9:15 am, Andy Johnson <andyjohns...@gmail.com> wrote:

> The event is used to tell the thread to exit, so it is logically part
> of the thread's 'purpose'. However, I take your point.

Fine, use an event that's queued to the completion port.

> If I understand it, your solution is that a second thread (b) waits on
> the event and wakes up the thread waiting on the completion port (a)
> by using PostQueuedCompletionStatus() to inject a completion packet.
> Presumably I would use a special value for the completion key to
> indicate that the event has signaled and that thread a should
> terminate. Is this correct?

Yes.

> I like this solution. The problem I have is that there is one 'exit'
> event, but many instances of thread a reading from the completion
> port. The thread b would therefore have to ensure that each reading
> thread exited properly (by sending multiple completion packets?).

Are you trying to terminate every thread or just some of them? If
you're trying to terminate every thread, just send as many packets to
the completion port as there are threads you need to terminate.

If this is a one-time thing that only happens on program termination,
you don't have to get this fancy. For example, you can check a global
termination flag before you block and you can timeout the IOCP check
every second. This will add a delay on termination, but that probably
doesn't matter.

> That, and the extra thread to watch the event, adds up to a lot of
> overhead. Intuitively, I feel that this should be simpler - but that
> may be my lack of experience.

You don't need the thread to watch the event. Instead of using the
event to signal termination, just send termination events to the IOCP.

> I feel that what I'm doing is probably quite common. Is this a
> standard implementation pattern for this problem?

It's hard to say without knowing more details. If all threads are
fully cooperating, and this is an unusual situation (say, the shutdown
of the entire application), then you have a lot of solutions. You can
use an APC to run termination code in the thread.

Frankly, the most common pattern for actual server termination is not
to bother shutting down the threads. You need to close down all the
connections they're handling anyway, and once there are no
connections, they will just block in GQCS forever, which is harmless.
The program can terminate with the threads in this state.

If you're trying to terminate an engine that's just part of an
application, common patterns include 'magic shutdown packets' you
queue to the IOCP and timeouts with a global flag that's checked just
before calling GQCS.

Use what makes sense in your app, but don't break normal thread
operation just for shutdown logic. It's not worth it.

DS

[Andy Johnson]

Thanks David. I really appreciate you taking the time to help me.

Andy

[Len Holgate]

> I feel that what I'm doing is probably quite common. Is this a
> standard implementation pattern for this problem?

Andy,

Personally I like to have things shut down cleanly, so I tend to use
the 'magic completion packet' method. I keep track of the number of
threads that are using the IOCP and then, when it's time to shut down,
simply post that many 'magic completion packets' to the port. Each
thread will wake, act on the packet and exit. I use a value of 0 for
the completion key, but there are probably other ways to do it...

If you need a bit more help then you could always take a look at my
free IOCP based server framework: http://www.lenholgate.com/archives/000637.html

Len