Reducing the default stack size in emscripten

[Sam Clegg]

I have an open PR to reduce the default stack size in emscripten from 5Mb to 1Mb, and we are also considering reducing it even furthur (possibly to 64Kb which is the wasm-ld default, or to 128Kb, which is the musl default): https://github.com/emscripten-core/emscripten/pull/14177.

How many folks out there have run into stack limits with the current limit of 5Mb?  How many folks are worried they would run into limits if we reduce the default to 1Mb, 128Kb or 64Kb?   Would those who feel they need more stack be OK adding `-sTOTAL_STACK` to their link command to request a higher limit?  (feel free to respond there, or on the issue above).

cheers,

sam

[Floh]

My C coding style is a bit stack heavy because it heavily prefers using the stack instead of heap allocations.

Example:

https://github.com/floooh/sokol-samples/blob/b7f8968f6dfd223a80b5ce9f50c40c2b79492403/sapp/cube-sapp.c#L19-L105

1 MB is quite certainly ok (but might be a problem for code with lots of recursions?), but I think 64 KByte is asking for trouble ;)

As long as there are clear runtime error messages which indicate stack overflow I'm fine with any size though, what would suck is random memory corruption caused by an overflowing stack.

Cheers,

-Floh.

[Jukka Jylänki]

+1 for a lower default. 64KB would be fine for me. The rationale for
that is that it should be a good practice for developers to be aware
that the stack length is fixed during runtime. Assuming we do have
good stack checkers enabled by default by now, then we should be very
good here.

It is good to note that 64KB wasm stack is not the same as a 64KB
native stack, because in wasm, function call stack and
non-struct&non-array arguments are not part of the Emscripten-compiled
stack. So if one has deep nested calls or recursion, that will not
consume the wasm stack. I'd expect that a 64KB Wasm stack will be like
a 128KB native stack, though of course varies on the application.

Btw, Floh in that code snippet, it would be good to change the
'vertices' and 'indices' arrays to be 'const' so that the compiler
will have an easy time to avoid placing them on the stack. That would
be excess work if it did, since it would need to memcpy a new memory
block copy every time the function is entered (to prepare for changes
in the values). Instead, when the compiler knows the values will not
be modified, it can retain a single copy of the data in the global
data section, avoiding use of the stack. It might do that already if
it can reason about the values not being modified, but that probably
depends on what the signature of sg_make_buffer() function is.

Back in 2019 in
https://github.com/emscripten-core/emscripten/pull/10019 I wanted to
reduce it to 512KB as the default, but that was a bit objectionable
then. Hopefully the issues have been resolved now, I'd love to see a
64KB default stack.

It would be nice to be conservative by default, and have developers
have to ask to get more, versus by default consuming more, and have
developers need to gain awareness in order to optimize to less. The
latter leads to suboptimal code being distributed on the web (probably
99% of Emscripten compiled pages out there are running with a 5MB
stack), the first leads to developer either being happy with 64KB
since most really don't use that much, or crashing once, and then
bumping the size (or changing their algorithms to avoid as much of the
stack). Imo having devs crash once to educate is better than
by-default suboptimal code.

Also a middle ground - "let's be a little suboptimal but not by much"
feels like it could be a worst of both worlds: it causes excess memory
usage on deployed pages still, but still won't probably save devs from
crashing once to learn. So super constrained seems like the nicest
default to me.

> --
> You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/0ef02392-4a8b-4442-a48c-957570977659n%40googlegroups.com.

[Sam Clegg]

Jukka, I wonder what you think about making `-Wl,-stack-first` the default for non-optimizing builds?  This will allow stack overflow to instantly wrap/trap rather than corrupting static data first.   It does mean the opt builds will have a different memory layout, but this doesn't seem particularly different to all the other things that are already different with opt builds.   Floh too, any thoughts on that?

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CA%2B6sJ-0Pd9WYDd95jnXFEONgfySWYUK5vVOCneTyFOo42c6C_Q%40mail.gmail.com.

[Floh]

> Btw, Floh in that code snippet, it would be good to change the
> 'vertices' and 'indices' arrays to be 'const' so that the compiler

> will have an easy time to avoid placing them on the stack...

Does the compiler actually this sort of magic with const? I was only aware

of static. But yes, real world code wouldn't place such data on the stack.

In the samples I just wanted to make clear (and also test) that the initialization

data doesn't need to stick around for the lifetime of the buffer, instead it is 

copied into the buffer (or image) inside the sg_make_xxx calls.

As for 64 KByte as default, I'm a bit divided. On one hand I agree with

you guys that it's better to have a small default which breaks early instead of a big default

which wastes memory for 99% of projects. On the other hand, platform-specific

compiler/linker options are a bit of a hassle in cross-platform build scripts.

I'd prefer a "useful default" which is "good enough" for many projects. E.g. I can already

see the "flood" of issues asking why the code using my libraries causes errors

in WASM even though they did everything "exactly like in the sample code" ;)

(after all, it looks like the default initial memory is 16 MBytes, and 64 KBytes of

16 MBytes is just 0.4% (if my math is right), IMHO a bit more "waste" for the stack

would be acceptable)

Cheers!

 

[Floh]

>  This will allow stack overflow to instantly wrap/trap rather than corrupting static data first.

This would be perfect IMHO. Why not also do this for optimized builds, are there performance problems with placing the stack at the bottom of the heap?

[Sam Clegg]

On Wed, May 19, 2021 at 9:25 AM Floh <flo...@gmail.com> wrote:

>  This will allow stack overflow to instantly wrap/trap rather than corrupting static data first.

This would be perfect IMHO. Why not also do this for optimized builds, are there performance problems with placing the stack at the bottom of the heap?

Its a binary size thing.  If static data can live at addresses < 65k  then any instructions referencing addresses within that data is more compact due to LEB encoding of the `i32.const <address>`.   Apparently its enough to make a difference.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/135251cf-08b6-46d6-a762-2f68629ae1b8n%40googlegroups.com.

[Sam Clegg]

Sounds like you would be more amenable to what my PR is currently doing which is reducing to 1Mb?

 

Cheers!

 

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/71303cf3-2d27-4dfa-b17c-bf000edddca6n%40googlegroups.com.

[Shachar Langbeheim]

What are the benefits of decreasing the stack size to 1Mb? If I understand correctly, the reduced binary size only happens if the stack size is 64K or less.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAL_va2-MeYutvr6j-zvQqEDdQbj-Hw%2BJnBwNHS5vcnJxyNxR5g%40mail.gmail.com.

[Floh]

> Sounds like you would be more amenable to what my PR is currently doing which is reducing to 1Mb?

Yep, I think 1MB or even 0.5MB sounds like a good middle round to me (at least from my own coding experience). I think really minimal projects would reduce INITIAL_MEMORY anyway instead of going with the default settings, and while at it, also tinker with the stack size.

[Floh]

> What are the benefits of decreasing the stack size to 1Mb

I don't know what the actual motivation is, but the current defaults of INITIAL_MEMORY=16MB and TOTAL_STACK=5MB look a bit "uneven" to me (e.g. of the initial allocation of 16MB, one third is already "wasted" for the stack.

[Floh]

> If static data can live at addresses < 65k  then any instructions referencing addresses

Oh, that's also good for me to know, because my coding style also makes heavy use of static global data embedded in the executable :)

[Nagy Imre]

Hi Sam,

We have a pretty big project and it is possible that we run into issues if stack size is minimalized.
We are OK, if the default stack size could be set with -sTOTAL_STACK.

Thank you,
Imre

--

You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/0ef02392-4a8b-4442-a48c-957570977659n%40googlegroups.com.

[Jukka Jylänki]

ke 19. toukok. 2021 klo 19.23 Floh (flo...@gmail.com) kirjoitti:
>
> > Btw, Floh in that code snippet, it would be good to change the
> > 'vertices' and 'indices' arrays to be 'const' so that the compiler
> > will have an easy time to avoid placing them on the stack...
>
> Does the compiler actually this sort of magic with const? I was only aware
> of static. But yes, real world code wouldn't place such data on the stack.
>
> In the samples I just wanted to make clear (and also test) that the initialization
> data doesn't need to stick around for the lifetime of the buffer, instead it is
> copied into the buffer (or image) inside the sg_make_xxx calls.

Yeah, const should make the compiler implement that optimization.
Using static should work too.

> As for 64 KByte as default, I'm a bit divided. On one hand I agree with
> you guys that it's better to have a small default which breaks early instead of a big default
> which wastes memory for 99% of projects. On the other hand, platform-specific
> compiler/linker options are a bit of a hassle in cross-platform build scripts.

I don't think this should be that bad. In this case, there already
exist compiler/platform specific flags for this, there is no standard.

Visual Studio uses /STACK:foo
GCC on macOS uses -Wl,-stack_size,0x100000000
https://stackoverflow.com/questions/2275550/change-stack-size-for-a-c-application-in-linux-during-compilation-with-gnu-com
Looks like Clang also uses the same
https://stackoverflow.com/questions/18909395/how-do-i-increase-the-stack-size-when-compiling-with-clang-on-os-x
MinGW on Windows uses -Wl,--stack,4194304 https://stackoverflow.com/a/2275586

(Adding support for that exact flag in Emscripten could be awkward,
since we'd have to parse it off instead of giving it to the linker,
but maybe in the future wasm adds an option to control the actual wasm
execution stack size, then we'd be stuck in conflict)

I would recommend dropping the stack size to at least as small as
512KB if this comes down to finding a compromise. Windows programs use
a 1MB stack by default, and in comparison our stack will not contain
function arguments, return values, or return addresses, and not even
function local data; only the data that have their addresses taken
(indexed by address/array) will be there. In practice the stacks
Emscripten programs utilize can be much smaller.

> I'd prefer a "useful default" which is "good enough" for many projects. E.g. I can already
> see the "flood" of issues asking why the code using my libraries causes errors
> in WASM even though they did everything "exactly like in the sample code" ;)

That is why we need to pair it with a meaningful error report. We
default to 16MB heap size by default, and pair it with a message to
enable memory growth or increase INITIAL_MEMORY - people do not report
bugs on that, and there are *lots* of users (most?) who need to
increase the memory size of their apps.

[Jukka Jylänki]

ke 19. toukok. 2021 klo 18.03 'Sam Clegg' via emscripten-discuss
(emscripte...@googlegroups.com) kirjoitti:

>
> Jukka, I wonder what you think about making `-Wl,-stack-first` the default for non-optimizing builds? This will allow stack overflow to instantly wrap/trap rather than corrupting static data first. It does mean the opt builds will have a different memory layout, but this doesn't seem particularly different to all the other things that are already different with opt builds. Floh too, any thoughts on that?

In non-optimized builds, those are not the tradeoffs: the difference
is not to trap vs corrupt static data, but to trap vs give a stack
overflow error, since we do have stack checks? (have had for a long
time, in two different performance modes even?)

Nevertheless, that would be fine for non-optimized builds. I do think
having the global data first is useful for code size for optimized
builds.

Pthreads and PROXY_TO_PTHREAD won't be able to benefit from this, but
it would give some added guard at least.

> To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAL_va29xhZYBDKbV__4u%2Bi%2BRfte7kMZuy-n%2BNX6AKCQynTx8cA%40mail.gmail.com.

[Jukka Jylänki]

to 20. toukok. 2021 klo 11.05 Shachar Langbeheim (nih...@gmail.com) kirjoitti:
>
> What are the benefits of decreasing the stack size to 1Mb? If I understand correctly, the reduced binary size only happens if the stack size is 64K or less.

Currently global data comes before the stack, so independent of what
the stack size is, we get the reduced binary size.

[Jukka Jylänki]

ke 19. toukok. 2021 klo 11.27 Floh (flo...@gmail.com) kirjoitti:

> 1 MB is quite certainly ok (but might be a problem for code with lots of recursions?), but I think 64 KByte is asking for trouble ;)

Btw, I would recommend you to try how small you can go. I would be
surprised if you are running into issues with 64KB today. (recursions
won't increase to this limit, unless the recursing functions have
locals that have their addresses taken)

[Maksim Ivanov]

One question: Will this planned change only affect is only for the main thread? In case there's no change for background threads (e.g., created via pthreads), it'll be useful to mention it somewhere.

[Sam Clegg]

Bumping this discussion because I'm taking a look at landing this once again.

In answer to the pthread question: Yes I'm planning on changing the default for both the main thread and pthreads (The plan is to make them the same by default).

--

You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/d7c00f7e-6bc2-4ae8-ae2d-0353ae4cff80n%40googlegroups.com.

[Floh]

My C code is still quite stack heavy, because a lot of my code essentially uses the stack as per-frame arena allocator ;) But I guess as long as there's a compiler/linker option to set the stack size I think it's fine.

Will a stack overflow cause a proper runtime error, or will this result 'undefined behaviour'?

[Sam Clegg]

On Tue, Oct 11, 2022 at 12:13 AM Floh <flo...@gmail.com> wrote:

My C code is still quite stack heavy, because a lot of my code essentially uses the stack as per-frame arena allocator ;) But I guess as long as there's a compiler/linker option to set the stack size I think it's fine.

Will a stack overflow cause a proper runtime error, or will this result 'undefined behaviour'?

The plan is to enable `STACK_OVERFLOW_CHECK=2` in debug (-O0) builds (currently we just use `STACK_OVERFLOW_CHECK=1`).   This should give precise errors at the point of overflow.

On Tuesday, 11 October 2022 at 01:34:07 UTC+2 s...@google.com wrote:

Bumping this discussion because I'm taking a look at landing this once again.

In answer to the pthread question: Yes I'm planning on changing the default for both the main thread and pthreads (The plan is to make them the same by default).

On Fri, Jul 9, 2021 at 3:24 AM 'Maksim Ivanov' via emscripten-discuss <emscripte...@googlegroups.com> wrote:

One question: Will this planned change only affect is only for the main thread? In case there's no change for background threads (e.g., created via pthreads), it'll be useful to mention it somewhere.

On Thursday, May 27, 2021 at 2:01:11 PM UTC+2 jj wrote:

ke 19. toukok. 2021 klo 11.27 Floh (flo...@gmail.com) kirjoitti:
> 1 MB is quite certainly ok (but might be a problem for code with lots of recursions?), but I think 64 KByte is asking for trouble ;)

Btw, I would recommend you to try how small you can go. I would be
surprised if you are running into issues with 64KB today. (recursions
won't increase to this limit, unless the recursing functions have
locals that have their addresses taken)

--

You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/d7c00f7e-6bc2-4ae8-ae2d-0353ae4cff80n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/7c70d4dc-d14b-4282-9a82-1c1e08393dd0n%40googlegroups.com.

[Floh]

PS: what would be *really* nice is a way to query the current and 'high water mark' stack sizes (only in debug mode, or with a specific build option), to get an idea how much stack a program/thread is using, instead of depending on trial-and-error.

[Sam Clegg]

On Wed, Oct 12, 2022 at 2:22 AM Floh <flo...@gmail.com> wrote:

PS: what would be *really* nice is a way to query the current and 'high water mark' stack sizes (only in debug mode, or with a specific build option), to get an idea how much stack a program/thread is using, instead of depending on trial-and-error.

The current value is accessible via ` emscripten_stack_get_current` defined in `emscripten/stack.h`: https://github.com/emscripten-core/emscripten/blob/main/system/include/emscripten/stack.h

Tracking the high water mark could be done with same binaryen pass that currently implement STACK_OVERFLOW_CHECK.. but it's not something we do at this point.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/218eec81-edce-45b7-b061-a1d57f381d3an%40googlegroups.com.

[Sam Clegg]

emscripten_stack_get_current is also designed to be super fast (its basically just a single `global.get` instruction).

[Sam Clegg]

Bumping this once again because I'm hoping to land the default stack size reduction change soon: https://github.com/emscripten-core/emscripten/pull/18191

In the end I decided to put the stack first in debug builds which gives a nice, consistent, error message on stack overflow: https://github.com/emscripten-core/emscripten/pull/18154

We also decided to go with 64KB default to match wasm-ld and wasi-sdk defaults.

I'll post again here once/if it lands.

cheers,

sam

[Sam Clegg]

3.1.27 has now been released and includes the reduction in the default stack size.

We do expect some folks to be affected by this and have to add `-sSTACK_SIZE=XXMB` to thier link flags.   If the effect is very widespread we can reconsider the rather drastic reduction.

cheers,

sam

[Floh]

I checked with my stuff, and the impact is much smaller than I expected.

- sokol-samples, doom-sokol and the 6502 and Z80 netlist simulations all work with 64KB stack (I would never have expected this tbh):

    https://floooh.github.io/sokol-html5/

    https://floooh.github.io/doom-sokol/

    https://floooh.github.io/visual6502remix/

    https://floooh.github.io/visualz80remix/

- my home computer emulators need to be bumped to 128KB stack (I'm probably doing something dumb in the initialization):

   https://floooh.github.io/tiny8bit/

I think I'll settle for a default stack size of 512 KBytes via a configurable cmake option in my build system wrapper.

Cheers!

[Brion Vibber]

I did a quick test on my ogv.js (with audio/video codecs built with emscripten) and found no obvious problems with libopus, libvpx (VP8/VP9), or dav1d (AV1) codecs on the new default. So far so good!

-- brion

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAL_va2-tLd_%3Dbsx86g8gqS%3Dg%3D_WRk_UjRZEEau2WmkPFMv1K9w%40mail.gmail.com.

[Sam Clegg]

Thank Brion, Floh,

So far it looks like minimal impact from this change.

cheers,

sam

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAFnWYTm8Xa1SpOKgMDFJ0W8a1YGyJrbiMhCMF2A6t1Xg18V_RQ%40mail.gmail.com.

[Steve Dekorte]

How about adding an API like:

Emscripten_extendStackIfNeeded(callback), which could be inserted anywhere stack depth might be an issue and would launch another coroutine if the stack was almost used up, swap to it, and swap back on return or exception?

[Sam Clegg]

On Fri, Dec 16, 2022 at 2:46 PM Steve Dekorte <st...@dekorte.com> wrote:

How about adding an API like:

Emscripten_extendStackIfNeeded(callback), which could be inserted anywhere stack depth might be an issue and would launch another coroutine if the stack was almost used up, swap to it, and swap back on return or exception?

Interesting, auto-magic, segmented and growable stacks.   I don't know of any platform that does this, but it is an interesting idea.  

I think it could be a lot harder than at first glance.  The biggest problem is that I think it would involve injecting checks everywhere in the wasm binary where SP is set and everywhere it gets restored.  Each of those locations would likely also need some kind of extra local state (e.g. previous segment pointer).   So maybe not impossible, but certainly not easy or free of runtime code.

Luckily, since the execution stack is completely separate and managed by the VM I don't think it would need to involve any kind of coroutine or control flow primitive.

 

On Tuesday, May 18, 2021 at 1:21:01 PM UTC-7 s...@google.com wrote:

I have an open PR to reduce the default stack size in emscripten from 5Mb to 1Mb, and we are also considering reducing it even furthur (possibly to 64Kb which is the wasm-ld default, or to 128Kb, which is the musl default): https://github.com/emscripten-core/emscripten/pull/14177.

How many folks out there have run into stack limits with the current limit of 5Mb?  How many folks are worried they would run into limits if we reduce the default to 1Mb, 128Kb or 64Kb?   Would those who feel they need more stack be OK adding `-sTOTAL_STACK` to their link command to request a higher limit?  (feel free to respond there, or on the issue above).

cheers,

sam

--

You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/4444d68e-5d77-448c-9e97-2cf11e8f0e09n%40googlegroups.com.

[Steve Dekorte]

FWIW. the C implementation of my scripting language (Io) does this and it worked well. IIRC, it was also used in PL/I. I've ported Io's C Coroutine implementation to emscripten fibers and, so far, it seems to work too. I should write some tests for this when I get a chance. One killer app of small stacks is for servers handling large numbers of sockets. Coroutines make this possible without having to implement buggy and inscrutable stack machines on top of callback hell. With dynamic stack sizes you get scalability without fragility, and without much overhead if the check locations are chosen carefully. Io checks the remaining stack size on each (Io level) block/method activation. As long as emscripten provided the API, developers could judiciously choose where to put the checks in their C code if they choose to compile their app with a smaller stack size. Some emscripten define for the stack size might be helpful there, if there isn't already one.

[Jukka Jylänki]

Most native OSes auto-grow the stack in native code. This is "easy" for them to do because they are able to leverage virtual memory and have a large address space, where a custom address range for the stack can be isolated. The way it is done is that the stack is grown in multiples of hardware pages, and after the end of the currently used stack, the pages are not mapped, which leads to a page fault being raised when an application tries to push the stack too much. At that point, the stack is then automatically grown inside the page fault handler. What this scheme gives you is that the hardware MMU is effectively then performing the safety checks in a zero cost manner.

In wasm we don't have either virtual memory with page fault handler support, nor a large address space like native programs have. Hence supporting automatic stack growth would mean adding a costly stack bump check inside each function. Unfortunately the upcoming wasm64 or virtual memory plans don't cover this kind of use case either.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/d3e50a8a-713e-4ac7-abe2-c5ddd781d702n%40googlegroups.com.

[Floh]

> Most native OSes auto-grow the stack in native code. 

AFAIK "most" excludes Windows though right? As far as I remember Windows gives me a hard crash if I'm overflowing the stack size determined in the linker invocation.

[Jukka Jylänki]

Is that when running a 32-bit or 64-bit process?

According to https://learn.microsoft.com/en-us/windows/win32/procthread/thread-stack-size there is this kind of two-stage reserved vs committed machinery at play on Windows, but it reads like there is no attempt to unboundedly grow, but a hard limit is set at thread startup. In a 64-bit program, my understanding is that setting the reservation to e.g. a hundred megabytes of address range should be possible, and it is only committed over if any local stack frame actually needs it.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/f4dc650d-4ceb-4ebb-aabf-d875ca2007f2n%40googlegroups.com.

[Floh]

Good question... it's not too long ago that I dropped 32-bit support, so I remember a 32-bit issue. In any case. But I never used the MSVC /STACK linker option with two numbers, so I'm telling it the 'reserve' size (not the 'commit' size), so I guess that means it's not taking up that much physical memory, but when it hits the reserve capacity it's still a stack overflow error?

Well nvm, kinda off-topic for Emscripten :)