Customizing default c/c++ libraries (dlmalloc).

[Alecazam]

I haven't found any articles on customizing the build of the default libraries.  Default dlmalloc behavior is to abort on allocation failure and align allocations to 8 bytes.  That's not a good default strategy in general for any application.  How do you rebuild the libraries with the defines to modify this behavior?  These are the dlmalloc defines that I'd like to alter. 

#define MALLOC_ALIGNMENT ((size_t)16U)

#define ABORT       abort()

#define MALLOC_FAILURE_ACTION   errno = ENOMEM;

#define ABORT_ON_ASSERT_FAILURE 0

What/where are the build settings for libcxx and others?  Are these all -O3, or can you build -O1/2/3 versions of all of these and link them to appropriate builds?

[Alon Zakai]

See `tools/system_libs.py`, that controls building all the system libraries. Can add defines and change build opts there.

- Alon

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nicholas Wilson]

Aborting on failure is an excellent default strategy -- the contortions an application needs to go through to free memory in such conditions are extreme. How can you even analyse what memory your application can release if you can't even allocate any data structures to use in the analysis? Abort on memory exhaustion is the only sane default strategy.

Allocations could arguably be aligned to 4 bytes if you're very careful about how you place doubles in structures. On a normal OS, there are tons of libc internals that will have uint64_t members in structures (eg for file offset handling), so 8 byte alignment is normally required, but on emscripten I think you get away with putting those at odd multiples of 4 because of the emulated 64-bit arithmetic. That just leaves doubles which might need the strict 8 byte alignment, and they're unlikely to show up in library code.

Having said that, 8-byte alignment is a good default because it's guaranteed by the C spec! A conformant malloc() must return blocks aligned to the strictest requirement of any primitive type. Writing a double to a typed array on Arm will require that 8-byte alignment.

Nick

[Alecazam]

I don't think I'd mentioned 4-byte alignment.   I typically align to 16 bytes (as per the defines).  That way SIMD can be mixed in with non-simd structs without needed custom allocator everywhere.  iOS/OSX and Win 64-bit already follow this strategy, 32-bit Win and 32-bit/64-bit Linux and Android are still lagging behind on this to eek out memory and hold to legacy but at a price to code generation.

Abort on failure is an issue since Emscripten has a fixed size heap.  We have resources that can be purged from memory, but if the heap aborts on failure then we don't get null back from dlmalloc, and the app just exits.  It's a reasonable default strategy, but not a shippable one.

This is what I consider a reasonable strategy, where attempts are made to free up space.  This is a simplification of what I do.  Typically the browsers provide no feedback as to actual ram or vram, so you're forced into a try and fail strategy, but have fail and abort enforced by default doesn't allow this approach.

freeMemory(size);

ptr = malloc(size);

if (!ptr) {

  freeMemory(size);

  ptr = malloc(size);

  if (!ptr)

    abort();

[Alecazam]

I should add that a malloc failure can occur at any size.  It doesn't mean the heap is completely exhausted.  If I requested a 16MB contiguous block, then fragmentation may have prevented that, but after freeMemory it may succeed.   Without exception support too, you're stuck with trying to solve this either before the malloc or immediately after.

[Nicholas Wilson]

Good points, fair enough. We just increase the heap size at link-time in our builds, but I can certainly see that emscripten's memory management is going to cause trouble in real scenarios. It seems pretty brave to ship stuff on a platform where you have to have a malloc() wrapper like that!

Nick

On Friday, 12 September 2014 00:01:53 UTC+1, Alecazam wrote:

I don't think I'd mentioned 4-byte alignment.   I typically align to 16 bytes (as per the defines).  That way SIMD can be mixed in with non-simd structs without needed custom allocator everywhere.

[Alecazam]

We do that too, but ASLR and the continued prevalence of 32-bit browsers (on 64-bit OS no less) signficantly cuts into the available contiguous memory to Emscripten.   I hope at some point, Emscripten makes the jump to 64-bit addressable memory.  Even then, having no way to increase heap size (to start small and grow) will be a big limiter to the technology.   I can't see allocating 2^54 bytes at startup.  Thanks for the suggestions.

[Bruce Mitchener]

If the only thing that you want to customize from the default libraries is malloc, you can build your own copy of dlmalloc.c with whatever flags you want and those symbols should end up overriding those of the default library when the link happens.

 - Bruce