Loading and unloading of shared libraries

[paw]

# Subject: Loading and unloading of shared libraries

Hi emscripten team,

I am currently experimenting with loading and unloading shared libraries during runtime, this means I want to compile C/C++ code to asm.js that loads and unloads shared libraries during runtime on demand.

The C++ code uses `libdl` to load the libraries into the memory (with `dlopen`), to initialize the libraries (with `dlsym`) and to unload the libraries (with `dlcose`).

I am following the instructions as described in the [emscripten - wiki - linking - site](https://github.com/kripken/emscripten/wiki/Linking).

This means I compile all the libraries to side modules (`-s SIDE_MODULE=1`) and the executable to a main module (`-s MAIN_MODULE`), which loads and unloads the other libaries.

I add the side modules to the virtual file system as pre-loaded files as described in [emscripten - wiki - packaging files - site](http://kripken.github.io/emscripten-site/docs/porting/files/packaging_files.html).

My C/C++ code that loads and unloads libraries looks similar to:

```C++

// ...

// load shared library

void* moduleHandle = dlopen("path/to/someModule.lib", RTLD_NOW);

// initialize shared library

void* initSymbolHandle = dlsym(moduleHandle, "Initialize");

typedef shared_lib* (*initialize_function_type)();

initialize_function_type* initializeFunction = reinterpret_cast<initialize_function_type>(initSymbolHandle);

shared_lib sharedLibPointer = initializeFunction();

// use shared library

sharedLibPointer->use();

// ...

// de-initialize shared library

void* deInitSymbolHandle = dlsym(moduleHandle, "DeInitialize");

typedef void (*de_initialize_function_type)(shared_lib*);

deinitialize_function_type* deInitializeFunction = reinterpret_cast<de_initialize_function_type>(deInitSymbolHandle);

deInitializeFunction(sharedLibPointer);

// unload shared library

dlclose(moduleHandle);

```

My first question is:

1. Is the memory before loading the shared library identical to the memory after unloading of the shared library? This means is the shared completely completely cleaned from memory, including the symbols fetched during de-/initialization phases?

My concern is that the memory grows with each load and unload of a library and therefore the memory consumption grows during runtime.

I already experimented with this and also compiled to WebAssembly. I am currently planning to consider compiling to WebAssembly in the future. Therefore I would like to know:

2. Is there a significant difference in the implementation of WebAssembly regarding the un-/loading of shared libraries and the clean-up of the memory in comparison to the asm.js implementation?

During my WebAssembly experiments, I received the following error message using Chromium (for Firefox it worked):

```command line output

RangeError: WebAssembly.Compile is disallowed on the main thread, if the buffer size is larger than 4KB. Use WebAssembly.compile, or compile on a worker thread.

```

3. Is there an option to work around or fix this issue?

For my tests I used:

* emcc version 1.38.15

* Chromium Version 70.0.3538.102 (Official Build) Arch Linux (64-bit)

* Firefox version 63.0.2 (64-bit) for Arch Linux

My final question is:

4. In [emscripten - wiki - building projects - dynamic linking](http://kripken.github.io/emscripten-site/docs/compiling/Building-Projects.html#dynamic-linking) you mention that dynamic linking is still experimental. Is this still the case? Do you have any plans to lift this feature out of experimental?

Best regards,

paw

[Alon Zakai]

I don't think we guarantee that, in fact I think we do very little to clean things up there currently. To improve that we'd need to add more sophisticated tracking of which symbol arrived from where, which has not been a priority.

I've heard native implementations don't guarantee a full cleanup either, btw, but I'm not sure.

My concern is that the memory grows with each load and unload of a library and therefore the memory consumption grows during runtime.

I already experimented with this and also compiled to WebAssembly. I am currently planning to consider compiling to WebAssembly in the future. Therefore I would like to know:

2. Is there a significant difference in the implementation of WebAssembly regarding the un-/loading of shared libraries and the clean-up of the memory in comparison to the asm.js implementation?

There should not be a significant difference there. However, the implementations are different enough that perhaps they are noticeably different in memory usage, for example, in wasm dynamic linking shared libraries have no JS component, so that doesn't even need to be cleaned up.

During my WebAssembly experiments, I received the following error message using Chromium (for Firefox it worked):

```command line output

RangeError: WebAssembly.Compile is disallowed on the main thread, if the buffer size is larger than 4KB. Use WebAssembly.compile, or compile on a worker thread.

```

3. Is there an option to work around or fix this issue?

That's a limitation Chrome imposes, which the only workaround is to load dynamic libraries in an asynchronous way - which means not synchronous dlopen. There is a JS API for doing that (I think it's loadWebAssemblyModule), and there are some current PRs also working on it further, see https://github.com/kripken/emscripten/pull/7548

For my tests I used:

* emcc version 1.38.15

* Chromium Version 70.0.3538.102 (Official Build) Arch Linux (64-bit)

* Firefox version 63.0.2 (64-bit) for Arch Linux

My final question is:

4. In [emscripten - wiki - building projects - dynamic linking](http://kripken.github.io/emscripten-site/docs/compiling/Building-Projects.html#dynamic-linking) you mention that dynamic linking is still experimental. Is this still the case? Do you have any plans to lift this feature out of experimental?

We definitely want to make this a stable and reliable feature, and there are people working on that now.

- Alon

Best regards,

paw

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[paw]

Hi,

Thank you very much for your answers which confirm our assumptions.

Is there a time schedule when to make this feature stable and reliable?

Best regards,

paw

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.

[Alon Zakai]

I don't think we can point to a specific time, it depends on a lot of things, in particular on how many people help out and how much - help is always welcome and necessary.

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.