Prevent multiple fetch + initialize for side modules

[Alex St. Louis]

So I have two builds of my program, a game engine, one where it's a single 37mb wasm file (5mb br compressed).

With my game engine, I use all threads available on my machine via Navigator.hardwareConcurrency (32). This works fine, as only the .js files are duplicated and my game starts up on my machine within 1-2 seconds with ~700ms of that been fetching and initializing the .js and wasm file and the remaining been engine + game startup time.

Now, I've started experimenting with hot reloading parts of my engine, to do this I've decided to use shared libraries and this has produced about 120 libraries around a few hundred KB and a couple libraries that are 11MB, 17MB. This has made the total size a little larger, around 42mb (6mb compressed), the size is not the issue though.

Now that the code is split among many wasm modules, emscripten is loading up each library (125 modules or 42mb) * threads (32), resulting in 1.3gb of code been downloaded and instantiated.

Is there currently a way to do what I'm trying to achieve in an efficient way, or would I have to modify emscripten for this purpose?

Also, slightly related, I can understand the worker.js files needing to match the threads, but is there not a way around the fetching of the extra non worker .js files?

Thanks

[Alex St. Louis]

Adding this check if (!ENVIRONMENT_IS_PTHREAD) before preloadDylibs(); inside run() has appeared to have solved the issue for me? Now they are only loaded once and I've tested calling between libraries and main and everything looks to be working fine.

Edit: Looks like I've run into issues when calling functions on other threads... as expected I suppose.

So this cannot work for me as is, because 42mb -> 1.3gb is just not very worthwhile just to keep using shared modules.

I'm actually mostly interested in runtime dynamic loading game modules, rather than engine modules, and the initial 140+ engine modules could be compiled as one MAIN_MODULE, so another way I've thought about doing this is to just build all the engine modules as a single module with MAIN_MODULE=2 and then at runtime I can load additional modules using emscripten_dlopen();. This looks like it should work, although it adds an 8+mb function (__wasm_apply_data_relocs) to the 40mb main module, this just happens to be too large of a function Chrome and probably any other browser.

I guess this next question should probably have it's own thread and probably better suited to llvm, but would it not be possible to split __wasm_apply_data_relocs and __wasm_call_ctors into multiple functions if they are too large?

Is there any way this is possible today with my large module, if not, what would have to change to make this possible?