Setting the stack size - does it actually do anything?
Fletcher Dunn
wlink directives actually do? According to the documentation, they are
supposed to set the stack size. However, from what I can tell, they have
absolutely no effect under Win32.
The only option that appears to matter is the compiler -s option, which
enables/disables code generation to check for stack overflow:
If I turn on stack overflow checking, then I can pretty much grow the stack
as much as I want - greatly exceeding the value that I specified as a stack
size.
If I turn off stack overflow checking, then I am not guaranteed to have the
amount available that was requested. The program will crash if you allocate
too much at one time, even if this amount is significantly less that the
amount you requested from the linker. There's a very brief sample program
below that illustrates this. (I'm using the command line in Windows XP.)
Note that the program requests a 4MB stack size, but then it can't even
allocate 128k at one time.
I guess my main question is - is it necessary for me to suffer the
performance penalties of stack checking in order to guarantee that my
program will not crash if a function allocates a large amount of memory on
the stack at once?
Thanks,
Fletcher Dunn
----------------------
/*
Test program to show stack problem. Compile and build with the following
command:
wcl386 -s -k4000000 stack_overflow.cpp
-s disables stack overflow.
-k is supposed to set the satck size. But it doesn't really appear to
do anything.
*/
#include <stdio.h>
void crashMe() {
char bigBuffer[128*1024];
bigBuffer[0] = '\0'; // !KABOOM!
printf(bigBuffer); // Make sure the compiler doesn't optimize out our
code...
}
void main() {
printf("About to crash...\n");
fflush(stdout);
crashMe();
printf("Wow - we made it!\n");
}
Michal Necasek
> Could someone please explain what the wcl386 -k option or the "option stack"
> wlink directives actually do? According to the documentation, they are
> supposed to set the stack size. However, from what I can tell, they have
> absolutely no effect under Win32.
>
Well, are the two executables identical? If not, then there's
obviously *some* effect. But you have to realize that the stack size is
to some extent advisory and the OS may or may not override it. However,
I don't think that is actually relevant here...
> If I turn off stack overflow checking, then I am not guaranteed to have the
> amount available that was requested. The program will crash if you allocate
> too much at one time, even if this amount is significantly less that the
> amount you requested from the linker.
>
Your conclusion is incorrect. There is a difference between allocated
and committed memory pages.
I recommend the -sg option to your attention. Read its description in
the User's Guide. It is likely to shed some light on what you're seeing.
Michal
Fletcher Dunn
>the OS may or may not override it.
What? You mean Microsoft software would take my clear instructions and
interpret "what I really meant" rather than just doing exactly what I
said??? I can't believe it?!?! :)
Thanks for this swift reply. The "sg" option sounds like exactly what I
need. Thanks for pointing it out.
I do think there's a problem though - perhaps just in the documentation.
The documentation and the functionality are not in agreement. According to
the documentation, my program should have worked:
<quote>
The "stack=" linker option specifies how much stack is available for the
primary thread when an executable starts. The "commit stack=" linker
directive specifies how much of that stack is committed when the executable
starts. If no "commit stack=" directive is used, it defaults to the same
value as the stack size.
</quote>
Since I did not specify a "commit stack" directive, this would indicate that
the amount initially comitted is the same as the amount specified by the
"stack" directive. (I've reproduced the exact same results calling the
compiler and linker seperately.) In other words, the documentation would
lead me to believe that by specifying "stack=xxx", the specified amount of
memory is allocated and COMITTED. But clearly this is not the case.
Or, if the stack size directive is indeed just a "suggestion" - then maybe
this extremely important fact should be mentioned in the documentation of
that directive.
Thanks again for your help. Open Watcom is a great product. We never EVER
trust Visual C++ to compile our code correctly under optimization. We've
been doing our release builds with Watcom since 1995, and I think in that
time, we've had a total of 2 compiler bugs.
Michal Necasek
> Since I did not specify a "commit stack" directive, this would indicate that
> the amount initially comitted is the same as the amount specified by the
> "stack" directive. (I've reproduced the exact same results calling the
> compiler and linker seperately.) In other words, the documentation would
> lead me to believe that by specifying "stack=xxx", the specified amount of
> memory is allocated and COMITTED. But clearly this is not the case.
>
> Or, if the stack size directive is indeed just a "suggestion" - then maybe
> this extremely important fact should be mentioned in the documentation of
> that directive.
>
Unfortunately, I believe that different Win32 implementations behave
differently. I'm pretty sure Win9x and NT/W2K/XP are different in this
regard. I do not know off hand what exactly the behaviour is on each
specific version. That's why documenting this behaviour may be difficult.
Note that I highly doubt Windows would ever decide to make the stack
size smaller than requested, but I'm 99% certain it may make it bigger.
BTW I assume that the problems you experience(d) were in a single
threaded app, as in a MT app the stack/commit directives only apply to
the primary thread (the -sg option still applies in MT apps).
Michal
Rod Pemberton
"Fletcher Dunn" <fletch...@yahoo.com> wrote in message
news:dnssep$puv$1...@www-1.scitechsoft.com...
> Or, if the stack size directive is indeed just a "suggestion" - then maybe
> this extremely important fact should be mentioned in the documentation of
> that directive.
The stack size directive isn't a "suggestion" for other environments, like
PM DOS. I have a DOS directory search program which will fault immediately
if I forget to specify -k256k.
Rod Pemberton
Sanych
wcl386 -s -k4000000 test.c
or
wcl386 -k4000000 test.c
Fletcher Dunn
> differently. I'm pretty sure Win9x and NT/W2K/XP are different in this
> regard. I do not know off hand what exactly the behaviour is on each
> specific version. That's why documenting this behaviour may be difficult.
Even if it works under Win9x, people making Win32 applications expect them
to work on 2000/XP. Due to the way Windows 2000/XP behaves (which I
understand is really nothing more than a BUG in the operating system) this
switch is seriously bad news on Win32, even if it is perfectly valid on
other operating systems. I really think the documentation (or compiler
itself?) should warn you about this.
It's worth noting how Microsoft's compiler handles stack overflow checking
(which they call "stack probes"). Not because the compiler is a good one,
but because nobody knows Windows better than Microsoft. Microsoft's
compiler doesn't have an option corresponding to -s to completely disable
stack probes. They do have an option /Gs<size> which sets the size below
which stack probes will not be used, and if you set this to a very high
number you can effectively disable stack probles. If you just use /Gs, the
the default size value winds up producing the same result as Watcom's -sg.
Also interesting is the fact they their "full optimization" switch (/Ox)
implies /Gs. In other words, using Watcom switch terminology, in the
Microsoft compiler -ox implies -sg, whereas in the Watcom compiler, -ox
implies -s.
I'm not trying to make a mountain out of a molehill - I mean I've found my
solution and so I'm good now. However, I really think you should consider
mentioning this in the documentation, considering how common and sneaky this
problem is. Anybody who uses -s and has a function call that allocates more
than 4K on the stack at once is in danger of crashing on Win32. That
doesn't seem to be a particularly unusual or rare combination of
circumstances. What makes it even more common (and sneaky) is that the -s
switch is implied by -ox. Now I'm sure that plenty of people would use -ox
("enable maximum optimizations") even if they wouldn't use -s ("disable
stack overflow checking.") In fact, that's how I got stung by this.
Anyway, bottom line: using -s or -ox on Win32 is a bad idea. This seems to
me that it's relatively common, so it deserves to be mentioned in the
documentation.
> Note that I highly doubt Windows would ever decide to make the stack size
> smaller than requested, but I'm 99% certain it may make it bigger.
I think my sample program proves that Windows will, indeed, allocate fewer
comitted pages than you request.
Michal Necasek
> Even if it works under Win9x, people making Win32 applications expect them
> to work on 2000/XP. Due to the way Windows 2000/XP behaves (which I
> understand is really nothing more than a BUG in the operating system) this
> switch is seriously bad news on Win32, even if it is perfectly valid on
> other operating systems. I really think the documentation (or compiler
> itself?) should warn you about this.
>
It's not a bug. Both OS/2 and Win32 use the stack guard page
mechanism, which is IMO stupid, but I can see the rationale for it (UNIX
systems work differently).
> It's worth noting how Microsoft's compiler handles stack overflow checking
> (which they call "stack probes").
>
Stack probe is not necessarily the same thing as stack overflow check.
Does MSVC really do overflow checks?
> However, I really think you should consider
> mentioning this in the documentation, considering how common and sneaky this
> problem is.
>
Sneaky, yes. Common, no - if it was, I'd be hearing about it a lot
more often :)
> That doesn't seem to be a particularly unusual or rare combination of
> circumstances.
>
It's very unusual for anyone who programmed in environments like DOS
where stack space is very limited and kilobyte-sized objects tend not to
be allocated on the stack. For people who come from UNIX background this
may be very different.
> In fact, that's how I got stung by this.
> Anyway, bottom line: using -s or -ox on Win32 is a bad idea. This seems to
> me that it's relatively common, so it deserves to be mentioned in the
> documentation.
>
But where? The -sg switch already explains (IMO adequately) the
situation. The biggest problem with documentation is that people tend
not to read it at all, or only when they get in trouble.
It could be argued that the -ox switch should imply -sg for Win32
target, but first the exact treatment of stack sizes and commit sizes on
various Win32 implementation (Win98, WinMe, NT4, W2K, XP, Server
2003...) would have to be known.
I still think the programmer should be responsible for turning off
-sg, because it's up to the programmer to determine if stack probes are
needed at all. The programmer may specify that commit size == stack
size, and then probes are pointless.
It should also be possible to modify the default Win32 termination
handler (the one that dumps CPU register contents etc.) to detect the
case of missing stack probe (if there was a page fault within the stack
area, stack probe is probably in order) and report that fact.
> I think my sample program proves that Windows will, indeed, allocate fewer
> comitted pages than you request.
>
The question is which version exactly... as I mentioned earlier, the
behaviour *is* different between various Win32 implementations. I rarely
work with Win32 so I don't know the details. MSDN documentation leaves a
lot to be desired - it is incomplete and appears to be slightly
inconsistent (eg. there is talk of rounding requested stack sizes up in
1MB increments in some instances, but no mention of that in other
relevant places).
Michal
Michal Necasek
> The question is which version exactly... as I mentioned earlier, the
> behaviour *is* different between various Win32 implementations. I rarely
> work with Win32 so I don't know the details. MSDN documentation leaves a
> lot to be desired - it is incomplete and appears to be slightly
> inconsistent (eg. there is talk of rounding requested stack sizes up in
> 1MB increments in some instances, but no mention of that in other
> relevant places).
>
obtained with Dr. Watcom NT. Values are listed beginning at top of stack:
1) Executable with both stack size and commit size set to 512 bytes
0x02000 committed (8K)
0x01000 guard page (4K)
0x3D000 reserved (244K)
-------
0x40000 total (256K)
2) Executable with stack size set to 0x6D800 (438K) and commit size
set to 0x10000 (64K):
0x10000 committed (64K)
0x01000 guard page (4K)
0x5F000 reserved (380K)
-------
0x70000 total (448K)
From this I would extrapolate that W2K sets the stack limit to 256K
minimum, regardless of what the executable headers says. The OS does
however honor the commit size. Stack sizes >256K are rounded up to more
than a page boundary, perhaps 64K (could be determined exactly if
someone was enterprising enough).
Needless to say, this contradicts MSDN which either mentions nothing
about minimum stack sizes and rounding, or talks about rounding up to
1MB. There's no telling if this behaviour is affected by Service Packs
or not - it might well be. There might even be some registry setting for it.
Bottom line, experimentation is required to determine real behaviour
because documentation can't be trusted. That's why I'm not inclined to
put any detailed information into the OW documentation, because if MS
can't get it right, how do we know that we can?
I'd be curious to know what the results are on other Win32
implementations. My test program was trivial:
-------
#include <stdio.h>
void main( void )
{
printf( "Press Enter to continue..." );
getchar();
}
-------
I used the -k switch of wcl386 to control the stack size.
Michal