WaitForMultipleObjects with more than 64 objects
Chris Mullins
on more than 64 items. Specifically I have 96 events that I'm waiting on,
any of which should wake up my main control thread.
The problem is that WaitForMultipleObjects doesn't handle (according to the
documentation) more than MAXIMUM_WAIT_OBJECTS which is defined as 64.
Does anyone know of any elegant workarounds? I can certainly come up with
several brute-force solutions, but there has to be a clean way to do this...
--
Chris Mullins
Stuart Dunn
If NT and any file/Network/otherIO try looking into ansyIO with ReadFileEx
and co.
other than that you can poll on a sleep (v.bad)
or use two threads (bad)
or try to cunt down on the number of event you are waiting on at one time
(best)
Anyway what are you trying to achieve with this? Can a change of design
help?
"Chris Mullins" <cmul...@spamsucks.yahoo.com> wrote in message
news:824gu2$obd$1...@ffx2nh3.news.uu.net...
Alex Blekhman
IMHO, this Richter's workaround is quite elegant:
MSDN Library
Periodicals
Periodicals 1997
Microsoft System Journal
January
Win32 Q & A
>
> ...
Steven Schulze
Chris Mullins <cmul...@spamsucks.yahoo.com> wrote in message
news:824gu2$obd$1...@ffx2nh3.news.uu.net...
> I bumped into a problem with the WaitForMultipleObjects when trying to
wait
> on more than 64 items. Specifically I have 96 events that I'm waiting on,
> any of which should wake up my main control thread.
>
> The problem is that WaitForMultipleObjects doesn't handle (according to
the
> documentation) more than MAXIMUM_WAIT_OBJECTS which is defined as 64.
>
> Does anyone know of any elegant workarounds? I can certainly come up with
> several brute-force solutions, but there has to be a clean way to do
this...
I remember seeing an article in a magazine once that dealt with this same
problem. I don't remember the details, but briefly, it worked something
like the following, if I remember correctly:
A class allows you to add an unlimited number of events to it's internal
list. Internally, it uses one event per 64 events to "link" to the next
list of up to 64 events, etc. This way, say an external event from the
second batch of events gets signaled. Internally, a WaitForMultipleObject
gets signaled by this event. It then signals the one and only event
(belonging to the class) that you are waiting for externally with
WaitForSingleObject. You can then query the class to find out exactly which
even was signaled originally.
I think that's pretty elegant.
Steven Schulze
Concord, CA
Steven Schulze
Steven Schulze <stev...@pacbell.net> wrote in message
news:OBpdnKNP$GA.265@cppssbbsa05...
I forgot to mention - for each additional batch of 64 events, the class will
need to spawn a new thread to wait for those events using
WaitForMultipleObjects.
Steven Schulze
Concord, CA
Dan Evens
<OBpdnKNP$GA.265@cppssbbsa05>...
> A class allows you to add an unlimited number of events to it's internal
> list. Internally, it uses one event per 64 events to "link" to the next
> list of up to 64 events, etc. This way, say an external event from the
> second batch of events gets signaled. Internally, a
WaitForMultipleObject
> gets signaled by this event. It then signals the one and only event
> (belonging to the class) that you are waiting for externally with
> WaitForSingleObject. You can then query the class to find out exactly
which
> even was signaled originally.
>
I think the term for that is probably "cascade." It can be made
to work for many limited resources. For example, if you need
to contact a large number of nodes on a network, say N, but
each node can only have M direct connections, and M is less
than N. Then you can cascade it by making a special kind of
node that adds identifiers and passes through messages.
It gets a bit messy, but it gets around the problem. And it can
be made to work in multiple layers as well. If the number of
special nodes exceeds M, then you make a second layer of
cascaded special nodes. And so on.
To the original poster: As an alternate, is there some way you
can group the waiting objects in some logical fashion? That way
you could reduce the number of objects by having related
external events trigger the same object. For example, any time
a user makes some input that could send the same signal,
rather than having distinct signals for each user. (Or whatever
is producing the signals.) That of course depends on the signal
somehow having a way to determine which user sent it.
--
Dan Evens
Standard disclaimers etc. No spam please.
Chris Mullins
Stuart Dunn Wrote:
>
> What OS/kinds of events are you waiting on?
I'm using NT4 and waiting on Events.
> or try to cunt down on the number of event you are waiting
> on at one time (best)
> Anyway what are you trying to achieve with this? Can a
> change of design help?
I can certainly rearchitect my code to have the required behavior, but I'm
trying to see if there is a somewhat elegant solution to the problem first.
Alex Blekhman (in this same thread) pointed me at one of Richter's MSDN
articles that addresses this problem quite well. I'm going to give that
solution a try...
--
Chris Mullins
Andrew Tucker
: problem. I don't remember the details, but briefly, it worked something
: like the following, if I remember correctly:
The article was "Waiting for More Than 64 Objects" by Jason Clark in the
October 1997 issue of WDJ. The code is available online at
http://www.wdj.com/archive/0810
HTH
--
--
/* Andrew */
WWW: http://www.halcyon.com/ast
Email: a...@halcyon.com
Jamie Hanrahan
<cmul...@spamsucks.yahoo.com> wrote:
>I bumped into a problem with the WaitForMultipleObjects when trying to wait
>on more than 64 items. Specifically I have 96 events that I'm waiting on,
>any of which should wake up my main control thread.
>
>The problem is that WaitForMultipleObjects doesn't handle (according to the
>documentation) more than MAXIMUM_WAIT_OBJECTS which is defined as 64.
>
>Does anyone know of any elegant workarounds? I can certainly come up with
>several brute-force solutions, but there has to be a clean way to do this...
You should understand that the reason for the 64-object limitation is that
WFMO, for many objects, is EXPENSIVE. In NT terms, to enter the wait, a
wait block has to be allocated for every object, and each waitblock is
queued to the object you're waiting for and then cross-linked to the
thread. When any of those objects are signalled all those wait blocks
have to be dequeued, unlinked, and deallocated back to pool. All of that
happens at DISPATCH_LEVEL and all except the pool allocation and free
happens with the dispatcher spinlock held.
(WFMO with fAll == TRUE is even MORE expensive. Every time ANY of the
objects is signalled, all the others have to be checked. This all
happens, you guessed it, at DISPATCH_LEVEL with the dispatcher spinlock
held.)
These hacks (Richter et al) for implementing waits for more than 64
objects will work, and they count as "neat hacks", but they multiply the
overhead for every wait/unwait by a factor of at least four.
Don't do this. Don't even wait for 64 objects with any frequency.
Redesign.
--- Jamie Hanrahan, Kernel Mode Systems, San Diego CA
Windows NT/2000 driver consulting and training
http://www.kernel-mode.com/
I'd prefer replies, followups, questions, etc., in news, not via e-mail.
Dan Evens
<3849d1f9...@nntp.cts.com>...
> You should understand that the reason for the 64-object limitation is
that
> WFMO, for many objects, is EXPENSIVE. In NT terms, to enter the wait, a
> wait block has to be allocated for every object, and each waitblock is
> queued to the object you're waiting for and then cross-linked to the
> thread. When any of those objects are signalled all those wait blocks
> have to be dequeued, unlinked, and deallocated back to pool. All of that
> happens at DISPATCH_LEVEL and all except the pool allocation and free
> happens with the dispatcher spinlock held.
I'm a little clueless here. Is the expense in terms of memory overhead?
Or is it CPU time to set up these structures? Or is it CPU overhead when
the thing is waiting? Or in terms of CPU to deconstruct everything when
the waiting stops? Or is it something else and I'm *way* clueless?
And what does "the dispatcher spinlock held" mean? Sorry, I use
NT, I don't "look under the hood" any more than I absolutely have to.
Also, is the expense linear in terms of the number of objects?
The cross linking seems to indicate that it might be worse.
> Don't do this. Don't even wait for 64 objects with any frequency.
> Redesign.
I should think this good advice regardless of how quick or how
small the memory requirements of waiting for 64 objects. Having
more than a few distinct things to wait for indicates a "busy"
routine. If there are really that many distinct things that can
start up or hold back a routine, the routine is probably trying to
be too many things to too many clients. Or maybe it ought not
to be a routine that happens on a signal, but should be a library
routine that the signalling client can do on its own when it needs.
In other words, the routine is probably much too complex.
Jamie Hanrahan
>Jamie Hanrahan <j...@cmkrnl.com> wrote in article
><3849d1f9...@nntp.cts.com>...
>> You should understand that the reason for the 64-object limitation is
>that
>> WFMO, for many objects, is EXPENSIVE. In NT terms, to enter the wait, a
>> wait block has to be allocated for every object, and each waitblock is
>> queued to the object you're waiting for and then cross-linked to the
>> thread. When any of those objects are signalled all those wait blocks
>> have to be dequeued, unlinked, and deallocated back to pool. All of that
>> happens at DISPATCH_LEVEL and all except the pool allocation and free
>> happens with the dispatcher spinlock held.
>
>I'm a little clueless here. Is the expense in terms of memory overhead?
>Or is it CPU time to set up these structures? Or is it CPU overhead when
>the thing is waiting? Or in terms of CPU to deconstruct everything when
>the waiting stops?
CPU time to set up and deconstruct. The memory overhead is small, and
waiting threads consume NO cpu time, regardless of how complex the wait.
>And what does "the dispatcher spinlock held" mean? Sorry, I use
>NT, I don't "look under the hood" any more than I absolutely have to.
Well, practically speaking, it means that no thread preemption or
timesliceing can happen, regardless of thread priority, while WFMO or
SetEvent or etc. is grinding through these 64 "things". Even on a
multiprocessor system. Since lots of user interaction depends on the
scheduler running threads (including the mouse and keyboard input threads,
which are part of NT, but are scheduled like any other threads), doing
WaitFor(Many)Objects too often may result in apps' (ALL apps, not just the
one doing the wait) not getting to run when they otherwise could. This
could show up as sluggish response to user input, excessive retries at the
network protocol levels, etc., etc., etc.
The limit of 64 thread handles to WFMO was chosen to minimize this impact.
(It's pretty much an arbitrary number, and if you patched the routine that
implemented WFMO you could use a larger number; there isn't a table of the
things you're waiting for that's allocated to 64 entries, or anything like
that.) But if a thread (or a set of threads) are doing
WaitFor(Many)Objects very very often, you're going to see some of this
degradation, even with "only" 64 objects.
>Also, is the expense linear in terms of the number of objects?
>The cross linking seems to indicate that it might be worse.
No, it's linear for nObjects >= 4. It doesn't take any more time to add
to or remove from the head or tail of a doubly-linked list, or the head of
a singly-linked list, regardless of the number of list entries.
For nObjects <= 3, the thread already includes three built-in wait blocks
that can be used to record the wait, so the pool allocation and free are
avoided; this is a significant improvement.
Details on all this are in _Inside Windows NT, 2nd Edition_, by David
Solomon.
>> Don't do this. Don't even wait for 64 objects with any frequency.
>> Redesign.
>
>I should think this good advice regardless of how quick or how
>small the memory requirements of waiting for 64 objects. Having
>more than a few distinct things to wait for indicates a "busy"
>routine. If there are really that many distinct things that can
>start up or hold back a routine, the routine is probably trying to
>be too many things to too many clients. Or maybe it ought not
>to be a routine that happens on a signal, but should be a library
>routine that the signalling client can do on its own when it needs.
>In other words, the routine is probably much too complex.
Ayup!
Guillaume Landru
Guillaume.
Chris Mullins wrote in message <824gu2$obd$1...@ffx2nh3.news.uu.net>...
>I bumped into a problem with the WaitForMultipleObjects when trying to wait
>on more than 64 items. Specifically I have 96 events that I'm waiting on,
>any of which should wake up my main control thread.
>
>The problem is that WaitForMultipleObjects doesn't handle (according to the
>documentation) more than MAXIMUM_WAIT_OBJECTS which is defined as 64.
>
>Does anyone know of any elegant workarounds? I can certainly come up with
>several brute-force solutions, but there has to be a clean way to do
this...
>
>--
>Chris Mullins
>
>
Andrew E. Walldorff
redesigned, but how would you suggest redesigning this....
Say I have a service that receives data from connected mode sockets. The
limitation of 64 object in WFMO means I can have a maximum of 64 active
sockets. However, I have 250 clients. Any suggestions. Changing to UDP is
not an option.
Jamie Hanrahan <j...@cmkrnl.com> wrote in message
news:384bf1c5...@nntp.cts.com...
David Wright
> Say I have a service that receives data from connected mode sockets. The
> limitation of 64 object in WFMO means I can have a maximum of 64 active
> sockets. However, I have 250 clients. Any suggestions. Changing to UDP is
> not an option.
>
Rather than using events as your signal of IO completion, can you instead
redesign to use IO completion ports? (Only works on NT). This gets around the
limitation.
David Wright
dak...@hotmail.com
r...@recon.org
<david...@freewwweb.com> wrote:
>Rather than using events as your signal of IO completion, can you instead
>redesign to use IO completion ports? (Only works on NT). This gets around the
>limitation.
I'm a huge proponent of cports - use them all over the
place, particularly with sockets. However, they're not an
infinite resource either... there are hints in MSDN that
cports may have a limit on the number of cpackets which may
be pending within them. Let's say you have a huge number of
active sockets, all posting completions to a common cport at
a high frequency. What happens when an I/O completes and the
cport refuses to accept the cpacket?
One possibility: Limit the number of sockets/files/whatever
which are associated with a single cport. If you have more
objects, create multiple cports each with its own pool of
worker threads. I haven't tried this, but it would seem a
resaonable compromise between the "one thread per socket"
novice approach and the threat of lost completion
notifications on a single, too-busy cport.
Stuart Dunn
"Andrew E. Walldorff" <an...@rcs-incREMOVE.com> wrote in message
news:3847c...@nebula.superior.net...
> OK, I'm intrigued.... This seems all well and good if the system can be
> redesigned, but how would you suggest redesigning this....
>
> limitation of 64 object in WFMO means I can have a maximum of 64 active
> sockets. However, I have 250 clients. Any suggestions. Changing to UDP
is
> not an option.
>
>
ReadFileEx\WriteFileEx
IOCompletion Ports
Or if your doing other things anyway you could even poll with
HasOverlappedIoCompleted....
I perosonally don't like option 3 but most unix programmers are suck with it
so if your were thinking of going cross platform.....
Stuart Dunn
"Alex Blekhman" <tk...@yahoo.com> wrote in message
news:38462FAE...@yahoo.com...
> I can't see original post so I'll reply to this one.
>
> IMHO, this Richter's workaround is quite elegant:
>
threads to create cascaded events. Talk about overhead.
1 Thread per 64 objects
the cost of WFMO * lots (see the post by the kernel mode guy lower down the
thread)
extra events to notify the threads higher up the cascade.
Plus all the extra context switches + stack space.
I would call it a mess.
If you are going to create extra threads then you may as well split up the
function and have one or more threads waiting subset of the events.
But really you want to get rid of some of those events or at lest reorder
them to that you only wait on a small subset at any one point in time.
I did see this article before but purposly didn't meantion it because I
thought it was complete pants.
> MSDN Library
> Periodicals
> Periodicals 1997
> Microsoft System Journal
> January
> Win32 Q & A
>
> >
> > ...
> >
> > "Chris Mullins" <cmul...@spamsucks.yahoo.com> wrote in message
> > news:824gu2$obd$1...@ffx2nh3.news.uu.net...
Ziv Caspi
>I'm a huge proponent of cports - use them all over the
>place, particularly with sockets.
So perhaps you could comment on using AcceptEx with
WSARecv/WSASend -- Does it work reliably?
>However, they're not an
>infinite resource either... there are hints in MSDN that
>cports may have a limit on the number of cpackets which may
>be pending within them.
?
Could you post references? As NT uses linked lists
almost all over the place, that seems unlikely.
---------------------------------------------
Ziv Caspi
zi...@netvision.net.il
r...@recon.org
(Ziv Caspi) wrote:
>>However, they're not an
>>infinite resource either... there are hints in MSDN that
>>cports may have a limit on the number of cpackets which may
>>be pending within them.
>
>almost all over the place, that seems unlikely.
I looked around but couldn't find it. This was from a bit of
sample code in some past version of VC or MSDN. As I
(barely) recall, an I/O Completion Port call had a specific
test for (ERROR_NOT_ENOUGH_QUOTA == GetLastError()) in
addition to the tests for ERROR_IO_PENDING and
ERROR_SUCCESS. I didn't dig a lot deeper than that, but it
left me with the impression that there might be a limit on
the depth of a cport. I don't know if it's hardcoded
(doubtful) or based on available resources (more likely).
Joe Drago
Freudian-slip-esque typo?
(j/k)
--
-------------------
Joe Drago
hate spam jido over at slip.net
Remove "hate spam" and "over", then read out loud
{Insert cool .sig quote here}
sirwi...@my-deja.com
"Andrew E. Walldorff" <an...@rcs-incREMOVE.com> wrote:
> OK, I'm intrigued.... This seems all well and good if the system can
be
> redesigned, but how would you suggest redesigning this....
>
> Say I have a service that receives data from connected mode sockets.
The
> limitation of 64 object in WFMO means I can have a maximum of 64
active
> sockets. However, I have 250 clients. Any suggestions. Changing to
UDP is
> not an option.
I think you need to describe things better here. Are the sockets in
threads? What is it you're currently waiting on?
Asynchronous sockets would give you a Windows event to "wait" on
instead of a kernel object. Threads, all though possibly expensive at
this size, are trivial to use with blocking sockets and any waiting can
be reduced to a single kernel object (with a registration object for
events that occur that can be polled after the kernel object has been
signaled). There are numerous approaches to this, but they rely
heavily on *precisely* what it is you are attempting to do.
Sent via Deja.com http://www.deja.com/
Before you buy.
Jim M
>I guess noone else has a dirty enough mind to see the naughty
>Freudian-slip-esque typo?
>
>(j/k)
Well, I was wondering who was going to point it out first :-) Not me,
I've done some dodgy ones on usenet, so I'm not pointing no fingers...
Jim M
--
@ Derbyshire
Ben Hutchings
: I guess noone else has a dirty enough mind to see the naughty
: Freudian-slip-esque typo?
We might just be too polite to mention it on technical newsgroups.
--
Any opinions expressed are my own and not necessarily those of Laser-Scan.