Cannot enlarge memory arrays when allocating 1Mbytes buffer

[Ishaan Venkatesan]

Today I ran into the following exception after allocating an std::vector<char> of size 718924 in C++. The exception said,

"Cannot enlarge memory arrays. Either (1) compile with -s TOTAL_MEMORY=X with X higher than the current value 16777216, (2) compile with ALLOW_MEMORY_GROWTH which adjusts the size at runtime but prevents some optimizations"

Although I was able to fix the exception by setting "-s ALLOW_MEMORY_GROWTH=1" flag, it is still surprising that a memory block of less than 1M would cause the exception in the first place. In addition, I would prefer not to use ALLOW_MEMORY_GROWTH in order to maximize performance.

Interestingly, when I turned on "ALLOW_MEMORY_GROWTH" flag, the Chrome browser was able to handle the large array correctly, but IE9 threw the same exception. 

Any thoughts on the cause of these problems?

Thanks for your thoughts!

[Alon Zakai]

That 1MB allocation must have just pushed it over the top of the total memory usage exceeding 16MB.

If you see a difference between two browsers here, it is likely either a browser bug or a timing-related issue, as running out of memory in emscripten is managed entirely by emscripten (we just allocate a typed array and handle allocations inside it ourselves), not by the JS engine. In other words, the browser's memory management should not matter. If you can consistently reproduce the problem and can share a testcase, please file a bug.

- Alon

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ishaan Venkatesan]

Hi Alon,

I was able to replicate the bug with a test case. Attached is the code for your reference.

The code would build with Visual Studio 2012. When running WebApp project, you could enter the number of bytes you want to allocate, and a popup window would show up indicating whether the allocation (i.e. resizing an existing buffer) is successful or not.

Running the software using a Surface Pro 3 with 8 G of memory & Windows 8 Pro OS, here are the test results I got:

1. Without "-s ALLOW_MEMORY_GROWTH=1", Chrome would allocate 8,000,000 bytes successfully, but would fail when allocating 9,000,000 bytes

2. Without "-s ALLOW_MEMORY_GROWTH=1", IE9 would allocate 8,000,000 bytes successfully, but would fail when allocating 9,000,000 bytes

3. With "-s ALLOW_MEMORY_GROWTH=1", Chrome would allocate 14,000,000 bytes successfully. 

4. With "-s ALLOW_MEMORY_GROWTH=1", IE9 would would fail when allocating 14,000,000 bytes, although 9,000,000 bytes would work

5. With  "-s ALLOW_MEMORY_GROWTH=1", Chrome would allocate 200,000,000 bytes successfully. But after trying to resize the buffer to 200,000,001 bytes, the call would fail.

Please let me know if you need more information. Thanks Alon!

[Alon Zakai]

I'm not on Windows so I can't build these.

About 200,000,000 bytes, Chrome likely fails because it has actually run out of memory (the next size it wants is 256MB or 512MB, and maybe there isn't enough contiguous address space in the 32-bit process). And failures without memory growth are not surprising, if the total used memory reached the current size of memory.

Aside from that, the other failures are potentially surprising, which leaves just IE9 behavior. Likely IE9 bugs - how about other versions of IE, and Firefox? If IE9 is the only one to fail, likely their bug.

- Alon

--

[Ishaan Venkatesan]

Hi Alon,

I removed Windows-related code and attached the web application that should run on Linux & Mac. 

As to Firefox:

1 Without "-s ALLOW_MEMORY_GROWTH=1", memory allocation of 11,000,000 bytes would succeed (better than Chrome), but 12,000,000 bytes would fail

2. With "-s ALLOW_MEMORY_GROWTH=1", we could allocate 1G of memory without any problem (much better than Chrome).

Thus the surprising result is really IE as you've pointed out. This is a big concern though, as my application would often allocate 100-200 M of memories, and cross-platform support is important. 

Do you happen to know any work around? I noticed that when the Javascript sends a large file to the C++ side by calling FS.writeFile(), the code would not crash on all browsers even when we are not setting  "-s ALLOW_MEMORY_GROWTH=1".In the document here, it said that the default file system with Emscripten is actually implemented through memory (i.e. MEMFS). In other words, IE is able to handle large in-memory data (>500 M in my test) when the data is accessed through the file system API. I am wondering if it possible to allocate large blocks of memory in Emscripten by using MEMFS as the back-end? (not sure how)

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.

[Alon Zakai]

What do you mean by fail? What specific error do you see in the console? Is it the same across all of these? I would expect to see different ones, as if memory growth is on, the only possible failure should be that the browser runs out of memory, and that error message does not look like emscripten's "cannot enlarge memory arrays" notification.

- Alon

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

[Ishaan Venkatesan]

The screenshot of the error is attached for your reference. It shows that with "-s ALLOW_MEMORY_GROWTH=1", IE would fail when allocating 14MB of memory indicating "enlargeMemory" errors. Same error would happen with Chrome when allocating much bigger memories.

Thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Alon Zakai]

"This' is not a function object" looks like a weird bug in IE9. Does it happen in current versions of IE?

- Alon

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

[Ishaan Venkatesan]

It happens with the Current Version IE on the Windows 8 machine (IE 11, Version 11.0.9600.17631)

[Alon Zakai]

Is this perhaps the same as

https://github.com/kripken/emscripten/issues/3180

Does the patch there fix your problem on IE?

- Alon

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

[Ishaan Venkatesan]

The patch did fix the problem with IE. It would be nice if the next SDK could generate the proper code automatically. Thanks!

[Alon Zakai]

Thanks for the confirmation, I pushed the fix to incoming.

- Alon

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.