Lazy files load from filesystem without major changes in C++ code

[George Evmenov]

I recently ported big C++ application into webassembly and the only issue I have so far is filesystem's .data file size.

What I need is to load some resources only on request (fopen call or something similar) from the C++ code, but I can't change all places in C++ code where files are opened to make them use asynchronous load or some JS code.

What I've found is that there is FS.createLazyFile, but it works only in web worker. I tried dumb approach with moving whole generated .js file into worker and it fails with "ReferenceError: screen is not defined". As far as I understand my whole app can't be moved into web worker, because it does rendering. (note: I'm newbie in the web and js related stuff and might be missing something simple)

So my questions are:

- is it possible to load resources on demand without changing the C++ code that opens files? I'm ok with making main thread/render stop while it is loading.

- if first is not possible could it be possible to add files into filesystem after the app was started, from the C++ code? All files, that need to be added are located near the .html .js and other files.

[Sylvain Beucler]

Hi,

On 04/04/2020 18:37, George Evmenov wrote:

I recently ported big C++ application into webassembly and the only issue I have so far is filesystem's .data file size.

What I need is to load some resources only on request (fopen call or something similar) from the C++ code, but I can't change all places in C++ code where files are opened to make them use asynchronous load or some JS code.

What I've found is that there is FS.createLazyFile, but it works only in web worker. I tried dumb approach with moving whole generated .js file into worker and it fails with "ReferenceError: screen is not defined". As far as I understand my whole app can't be moved into web worker, because it does rendering. (note: I'm newbie in the web and js related stuff and might be missing something simple)

So my questions are:

- is it possible to load resources on demand without changing the C++ code that opens files? I'm ok with making main thread/render stop while it is loading.

I believe it'd be possible to implement a custom Emscripten filesystem that, leveraging ASYNCIFY, could pause execution, download the file on-demand and then resume.
I dropped this option for RenPyWeb (game engine web port), because that would make the whole application unresponsive in case of network issue during a fopen().
There's something in development called "asmfs" in emscripten that may be close, but it's not very clear what its goal is.

- if first is not possible could it be possible to add files into filesystem after the app was started, from the C++ code? All files, that need to be added are located near the .html .js and other files.

In RenPyWeb I initialize the FS with a small core .data, then download and extract an additional game.zip file, and then I download some other files in advance through emscripten_run_script, XMLHttpRequest and FS.writeFile({canOwn:true}) (to save memory).
See https://github.com/renpy/renpy/blob/master/renpy/webloader.py

emscripten_wget* functions may be more straightforward. I didn't use them because of various asynchronicity issues.

Beware of memory usage, and avoid LZ4 for your .data because that makes files read-only.

Hope this helps!
Sylvain

[George Evmenov]

Thank you for the responce.

There's something in development called "asmfs" in emscripten that may be close, but it's not very clear what its goal is.

"asmfs"  seems to be temporary disabled ( https://github.com/emscripten-core/emscripten/issues/9534 )

emscripten_wget* functions may be more straightforward. I didn't use them because of various asynchronicity issues.

 This is what I was looking for. I tried emscripten_wget, but it also require ASYNCIFY, which greatly increased my binary size (14Mb -> 21Mb) and as far as I understand this would also slow down whole application

> Asyncify adds overhead, both code size and slowness, because it instruments code to allow unwinding and rewinding. That overhead is usually not extreme, something like 50% or so.

So I'm using emscripten_async_wget now, without ASYNCIFY. It preloads all files into filesystem and then no other code changes are needed.

Beware of memory usage, and avoid LZ4 for your .data because that makes files read-only.

Should I worry about memory, if all files I have are ~100Mb total?

[Alex St. Louis]

What kinda of C++ application is it, George? Gabriel Cuvillier, the dev that did the Doom 3 demo port, used lazy loading to instantiate the initial download and get the user into the menu which is a small 15MB download.

I've been thinking a lot about heavy 3D graphics applications on the web with late code sizes, and fetching the assets in small chunks as needed is definitely the best option for now. Will be interesting to see with WASI if you'll be able to use your local hardware storage natively to download these big files too, which would be a hybrid web/desktop approach. Having some sort of file manager window in the canvas for big WASM applications is a good idea too I think.

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/90be231e-ae46-4b38-9446-0dcbfc431409%40googlegroups.com.