Support for externref

[ben layet]

Does emscripten support externref reference types for holding opaque handles to JS objects in C++? If not, any idea when?

[Thomas Lively]

Not yet, but there are some folks working on this. Perhaps it will be ready to use in a few months? It’s hard to say for sure.

On Sat, Jan 23, 2021 at 03:49 ben layet <lay...@gmail.com> wrote:

Does emscripten support externref reference types for holding opaque handles to JS objects in C++? If not, any idea when?

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/7f6178fa-bd05-4f65-aa7c-9e5a405bf851n%40googlegroups.com.

[ben layet]

Thanks!
Is there a feature ticket in a publicly visible ticket tracker that I can keep an eye on?

cheers

ben

[Thomas Lively]

I don't think so, but you can keep an eye on commits from "wingo" and "pmatos" on reviews.llvm.org to keep track of the progress.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/68c9b121-f12f-44fa-a91c-2a8b73d2f456n%40googlegroups.com.

[Karl Werner]

Does anyone have news on this topic?

[Thaina Yu]

Is this mean externref already usable?

https://github.com/emscripten-core/emscripten/pull/15913

[Thomas Lively]

Yes, but note that the usage in that commit is only in hand-written assembly code, not in C/C++.

More recently, support for externref landed upstream in clang: https://github.com/llvm/llvm-project/blob/main/clang/test/CodeGen/WebAssembly/wasm-externref.c. AFAIK we haven't yet experimented with it in Emscripten, but in principle it should work.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/57b1491a-7025-4967-a66e-ebcbf10f5e2bn%40googlegroups.com.

[Thaina Yu]

Do we have anyway to write hand-written assembly code along with emscripten? And is it possible to return that externref from wasm code into emscripten world?

I am actually using unity which use emscripten to build webgl. Would it possible to use EM_ASM_PTR to write wasm code to return as pointer (externref ? (an so I might return pointer as IntPtr in C#)) ?

[Sam Clegg]

On Thu, Mar 23, 2023 at 6:52 AM Thaina Yu <thai...@gmail.com> wrote:

Do we have anyway to write hand-written assembly code along with emscripten? And is it possible to return that externref from wasm code into emscripten world?

Here is the example how to use externref in assembly:  https://github.com/emscripten-core/emscripten/blob/main/test/core/test_externref.s

Until recently there was no way to communicate those references directly to C++ but it maybe possible now that upstream clan support has landed.

There is no current support for externref in EM_ASM.   We could potentially add EM_ASM_REF which returns a ref to C/C++ but until recently that would have been impossible.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/ed80516b-e1ac-4ba1-a535-acd1f632a2f5n%40googlegroups.com.

[Thaina Yu]

Thank you very much. Are there any public discussion or roadmap to add EM_ASM_REF into emscripten?

[Sam Clegg]

On Thu, Mar 23, 2023 at 9:20 AM Thaina Yu <thai...@gmail.com> wrote:

Thank you very much. Are there any public discussion or roadmap to add EM_ASM_REF into emscripten?

You are the first person that I know of to suggest it.

Be aware that using externref directly in C/C++ is very limited, even after the recent changes.  For example you can't store them in structs or class or store them to memory.  All you can really do with them is pass them around and (maybe) put them in tables.  So in practice I'm not sure how useful this stuff is yet, but I could be wrong?

 

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/e4853c85-e981-4fa4-a6d8-f64a7ad5c048n%40googlegroups.com.

[Thaina Yu]

I am actually using emscripten from unity and I mainly use C# code from unity side. But there are tons of 3rd party library and SDK that written with assumption that we will only use JavaScript. It then need us to construct javascript object in javascript and pass javascript object reference all over the places. For example, the google's firebase SDK that we need to construct it special object type to work with its complex services

It then become very frustrating to wrap everything in an API with mergeInto library that kept growing for everythings related to js object

With just the ability to pass externref into C# as IntPtr or int64 and so C# can pass it to another js API after it was done everything in C# is just what we really need, everything would come together as a flow of reference. The chain of bottleneck is that unity rely on emscripten to fully support externref first then it could start support it in some future. So the sluggish are stacking up

[Sam Clegg]

On Thu, Mar 23, 2023 at 9:51 AM Thaina Yu <thai...@gmail.com> wrote:

I am actually using emscripten from unity and I mainly use C# code from unity side. But there are tons of 3rd party library and SDK that written with assumption that we will only use JavaScript. It then need us to construct javascript object in javascript and pass javascript object reference all over the places. For example, the google's firebase SDK that we need to construct it special object type to work with its complex services

It then become very frustrating to wrap everything in an API with mergeInto library that kept growing for everythings related to js object

With just the ability to pass externref into C# as IntPtr or int64 and so C# can pass it to another js API after it was done everything in C# is just what we really need, everything would come together as a flow of reference. The chain of bottleneck is that unity rely on emscripten to fully support externref first then it could start support it in some future. So the sluggish are stacking up

But `IntPtr` and `int64` are fundamentally not externref, right?   Or are you imagining those being indexes into a table of externref perhaps?  That would work, but then the tricky part becoming managing that table.  This is mind what embind does when it refers to JS object from native code I believe.

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/1018024e-ab04-46c1-bb22-f73183601976n%40googlegroups.com.

[Thaina Yu]

I don't actually know which things unity would use to hold the value of externref, just guessing it would be IntPtr. But by anything it require that emscripten can officially define externref first anyway

[Sam Clegg]

On Thu, Mar 23, 2023 at 10:12 AM Thaina Yu <thai...@gmail.com> wrote:

I don't actually know which things unity would use to hold the value of externref, just guessing it would be IntPtr. But by anything it require that emscripten can officially define externref first anyway

The problem is that externref can't live in memory, so it can't live inside a struct or class or array or any C#/C++/C data type.  So it's not going to be very usable IIUC.   We already have `EM_VAL` in emscripten which is a way to refer to a JS object via an integer handle.   I don't think we can build anything better than EM_VAL any time soon, at least nothing that can work well with C/C++/C#, but perhaps I'm missing something?

  

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/8cb8c47e-fc87-4429-8dcd-1057c6f090f6n%40googlegroups.com.

[Thaina Yu]

I am not sure how limited it is but there are many situation that we just need a value to pass object from one function to another immediately in the same flow

By the way, which the handle would be returned as EM_ASM_INT or EM_ASM_PTR ?

[Sam Clegg]

On Thu, Mar 23, 2023 at 6:50 PM Thaina Yu <thai...@gmail.com> wrote:

I am not sure how limited it is but there are many situation that we just need a value to pass object from one function to another immediately in the same flow

By the way, which the handle would be returned as EM_ASM_INT or EM_ASM_PTR ?

 EM_ASM_INT is probably what you want if you want to return an integer handle to JS object.    e.g. Something like an EM_VAL or a handle to JS promise.  

In emscripten we use this thing called a HandleAllocator to map integers to JS objects:  https://github.com/emscripten-core/emscripten/blob/3cf83fdc4711e9bf3b01cf9821e717311a13c1b5/src/library.js#L3714-L3715

Because native code wants to deal with integers whenever we want to pass a JS object to native code we go through this handle allocator.   When JS receives an integer handle from native code it can turn it back into a JS object by looking it up in a given handle allocator.   In theory being able to pass externref directly looks like a way to remove the need for the extra mapping, but in practice C/C++ needs to be able to store handles in memory (and in data structures that live in memory) so using externref instead of `int` for EM_VAL doesn't work (at least not today).

To view this discussion on the web visit https://groups.google.com/d/msgid/emscripten-discuss/82954f49-c753-4c67-8de7-e5524f1e7f0en%40googlegroups.com.

[Thaina Yu]

Thank you very much