Guide for working with the heap?

[Marc Fawzi]

Hi,

Somehow when writing a large amount of data (over the 16MB default) to the Module heap (HEAPU8) I saw what could be a bug when ALLOW_MEMORY_GROWTH was set. Basically, when I print out the content I saw Emscripten’s own symbols, the C++ program symbols and a bunch of binary gibberish before the start of the the data I had written, as if my read offset was somehow smaller than my write offset. I’m working with someone else’ code and trying to understand what’s going on but when setting the memory manually at build time instead of allowing it to grow the code works just write and I get the exact data I write back when I read it. 

Any guides or blog posts out there for working with the heap?

Thanks,

[Alon Zakai]

Memory growth does mean you need to be careful when accessing the heap from JavaScript. Emscripten will update the HEAPU8 etc. views, but if you held on to references to the old views, or subarrays of them etc., those would become invalid, and you'd silently miss updates to memory. Basically, growth has the effect of creating replacements for all the JS memory views.

(This is even trickier with multithreaded code using memory growth.)

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CACioZivWDQ4HKHu6sdHq%2B%3Dyk2WX_GsiUcanTPDyNam1cWMmcUQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Marc Fawzi]

Oh, that does sound tricky, including in some single threaded scenarios, like the one below...

How would you modify the code below to avoid landing in invalid memory? I can take a guess but ... still learning about the JS side of it

function decompress(data / * Uint8Array */) {

 

    let dataPtr = Module._malloc(data.length);

    let dataHeap = new Uint8Array(Module.HEAPU8.buffer, dataPtr, data.length);

    dataHeap.set(data);

    

    let num_bytes_decomp= Module.C_getContentSize(dataHeap.byteOffset, data.length); 

    let decompPtr = Module._malloc(num_bytes_decom);

    let decompHeap = new Uint8Array(Module.HEAPU8.buffer, decompPtr, num_bytes_decomp);

    let len = Module.C_decompress(decompHeap.byteOffset, num_bytes_decomp, dataHeap.byteOffset, data.length);

    let result = new Uint8Array(decompHeap.subarray(0, len));

    Module._free(dataHeap.byteOffset);

    Module._free(decompHeap.byteOffset);

    return result

}

Thank you in advance

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpSDrHS89s-NxYStqn81RLD03Pg78Ns07QMc%3DMzdQzHKAw%40mail.gmail.com.

[Alon Zakai]

That code might actually work even with growth - since I think you just use dataHeap.byteOffset after the view might be invalid? I think the byteOffset would be unchanged.

But the safe way is to not create a view and refer to it after a call that might grow memory. Creating all new views after any possible growth is the safe thing.

In this example, you can just save the byteOffset, which is just an integer, to a local variable.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CACioZisfumAkg%3Dk92L5zCmPgmnCzx5-TOr9rW6OHwwXA528_Jw%40mail.gmail.com.

[Marc Fawzi]

Yes, thank you for explaining... I believe it would fix the problem. 

num_bytes_decomp is over 16MB and Module._malloc(num_bytes_decomp) was causing the memory to grow

I'm doing it differently now with much smaller value (low hundreds KBs) for num_bytes_decomp so I'm not using automatic memory growth anymore and going with the default memory size

Thanks a lot. Really appreciate it.

  

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/CAEX4NpREk6-PsvBv3rtOt3pW-MOgyD7VwXamnH%3DVEP37vwLtBA%40mail.gmail.com.