The future of embind...

[Dean Elhard]

Since 1.13.0 is moving to the fastcomp compiler, which only supports asm.js, and embind doesn't work with asm.js, it seems that anyone relying on embind is being left behind.

The issues I see on github regarding embind and asm.js are old and inactive, and indicate no plans to make embind work with asm.js any time soon.

Any comment on the future of embind (if any), or what will replace it?

[Chad Austin]

Hi Dean,

We should port embind to asm.js!  It's certainly a bit of work, but probably not too hard.  It's really just that nobody's done the work yet.  Since IMVU, my employer, can't use asm.js until it supports a growable heap (and perhaps closure compiler), we aren't terribly motivated to do the work.

That said, part of me just wants to bite the bullet and do the conversion myself if I can find the time.  :)

In case someone else wants to take a crack at it, here is what I know:

There are a couple places where embind uses reinterpret_cast on function pointers in order to pass more arguments than their prototypes specify.  For example, see https://github.com/kripken/emscripten/blob/master/system/include/emscripten/val.h#L67

Those functions need to be converted to varargs.  (At the time, I think we were blocked on a varargs issue with the old LLVM, but that may not be an issue anymore.)

The other issue is that embind relies on JavaScript being able to look up functions by indexing into FUNCTION_TABLE[x].  asm.js doesn't have a single function table: it has a whole bunch of function tables, one for each possible type signature.

Thus, on the C++ side, we need a way to take a C++ signature (like float(const void*, int&)) and turn that into a string ("fii") that we can use to select one of the function tables on the embind side.  Embind would also need the compiler to output a table from signature to the appropriate function table.  The compiler-generated glue would look something like:

var FUNCTION_TABLES = {

  vi: FUNCTION_TABLE_vv,

  fii: FUNCTION_TABLE_fii,
  ...
};

I think that if we had those two things, we could make embind work with asm.js.

*waves hands a bit* :)

Hope that's helpful,
Chad

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Chad Austin

Technical Director, IMVU

http://engineering.imvu.com

http://chadaustin.me

[Alon Zakai]

Note that fastcomp supports a growable heap in asm.js mode. It will not validate as asm.js, but should still get much of the speedups from it, just without guarantees and probably not all of them.

- Alon

[Chad Austin]

True!  We do want to use fastcomp but wasn't sure the advantages outweighed the embind porting work.

However, because Dean asked, I sat down and took a look tonight, and got a few functions working.  Most of the work is in getting rid of the function pointer casts, but it seems quite straightforward.

I'll poke away and send pull requests when I have them...  If anyone wants to help, let me know.  :)

[Dean Elhard]

I would be interested in helping, although I haven't looked at how it works, so I am not sure what exactly is involved...

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

Chad Austin

Technical Director, IMVU

http://engineering.imvu.com

http://chadaustin.me

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[Chad Austin]

I found some time and got a few bits of embind working:

https://github.com/imvu/emscripten/commit/afc59d1e331c445a5762f6a5751b86740b1ebbeb
https://github.com/imvu/emscripten/commit/228b4fc349f1cdf592821d988240e7d9543462be

When I get a minimal demo up and running in fastcomp/asm.js, I'll submit a pull request.

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[ILOVEPIE]

Clang already uses the Intel ABI for it's name mangling, which produces names for functions containing their parameter types, if we performed the embind on those names we would be able to figure out the parameter types.

On Tuesday, March 4, 2014 2:46:48 PM UTC-8, Chad Austin wrote:

Thus, on the C++ side, we need a way to take a C++ signature (like float(const void*, int&)) and turn that into a string ("fii") that we can use to select one of the function tables on the embind side.  Embind would also need the compiler to output a table from signature to the appropriate function table.  The compiler-generated glue would look something like:

var FUNCTION_TABLES = {

  vi: FUNCTION_TABLE_vv,

  fii: FUNCTION_TABLE_fii,
  ...
};

[Chad Austin]

I made substantial progress on getting emscripten::val to work in asm.js tonight.  Just emscripten::val, not any of the function or class binding stuff.

https://github.com/imvu/emscripten/commits/master

It no longer uses reinterpret_cast on function pointers.  Instead it uses varargs.

Unfortunately, my demo doesn't quite run yet.  The old compiler had 8-byte alignment on varargs, but the new compiler appears to usually fit integers in 4 bytes.  I'll need to find some way to switch off of whether asm.js is enabled in the embind JavaScript.

Alon: do you know if there's an easy way to detect whether fastcomp or asm.js is enabled at runtime?

Also, what _is_ the vararg convention for fastcomp/asm.js?  Is it the same as asm.js in the old compiler?  Or is fastcomp the new factor here?

[Alon Zakai]

In library.js we ifdef on that in a few places, see formatString for example. Yes, this changes by compilation mode. We used to do 64-bit alignment all the time, but in fastcomp we use the pnacl varargs lowering pass, which does only 32-bit alignment. We should fix that eventually as it means doubles are slower in varargs.

- Alon

[Chad Austin]

Oh, in fastcomp, it's always 32-bit alignment only?  I guess I assumed doubles would still be aligned.  Interesting.

The C++ (and C?) standard specifies that floats are converted to doubles in varargs, so 32-bit alignment could affect all floats and doubles.

I just realized I can work around the code generation differences here by doing my own packing into a void* args pointer with a variadic template.  I may try that next, as it makes embind more portable across code generation modes.

[Alon Zakai]

This is only in varargs, and due to the pnacl varargs pass. Fastcomp does align doubles inside structures properly and so forth.

Yes, this affects all floats in varargs, good point. Should be fixed.

- Alon

[Chad Austin]

I just submitted a pull request that makes emscripten::val compatible with fastcomp/asm.js.

https://github.com/kripken/emscripten/pull/2264

The technique used to achieve portability across both versions of the compiler is to use variadic templates to generate the varargs data, rather than relying on the compiler to do it.  https://github.com/imvu/emscripten/commit/aa05c2ee438ed1904ef90ebaebe3193dccd7f222

Next I will see about making the rest of embind compatible with fastcomp/asm.js.

[Warren Seine]

Thanks a lot Chad, I will test it this week.

[Chad Austin]

Just so you know, you'll have to modify emcc to disable the errors about using embind with fastcomp/asm.js.

Also, only the emscripten::val type works for now.

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chad Austin]

To keep everyone updated, this weekend, with Jukka's help, I managed to get embind running in fastcomp/asm.js.  The relevant pull request is: https://github.com/kripken/emscripten/pull/2287

(I also hope to get https://github.com/kripken/emscripten/pull/2286 which reduces code size for projects with tons of bindings.)

My little test program runs, but since IMVU doesn't use fastcomp or asm.js yet, I haven't been able to test against the IMVU code base.

Can someone remind me how to run the embind tests within the Emscripten test suite?

To answer your performance question:

If you want to reduce JS->C call overhead as low as possible, having JS call into C functions is probably best.

However, embind is certainly not slow.  Jukka did some work last year to dynamically create optimized functions when binding a C++ function to JavaScript.  I can't remember the exact amount of overhead relative to C, but in our experience at IMVU, embind overhead hasn't been a problem for us.

The main difference between embind and raw C function calls is that embind will assert that parameters have the same type and range.  For example, you can't pass an object to a function that's expecting a string and you can't pass the value 1000 to an unsigned char.

We've talked about having a mode where we disable parameter validation, but we simply don't do enough calls between JavaScript and C++ per frame for it to matter.

I encourage you to do your own measurements and see if they would affect your decision!

Cheers,
Chad

On Wed, Apr 9, 2014 at 11:23 AM, Joshua Litt <joshu...@google.com> wrote:

Thanks for doing this work Chad!  I have started an open source port of liquid fun(C++ physics library based on box2D) using embind.  I was terrified to read embind might be going away.

From a perf standpoint, is it better to just wrap the whole api in C, and then use normal emcc + handwritten js bindings?

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Chad Austin

Technical Director, IMVU

http://engineering.imvu.com

http://chadaustin.me

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alon Zakai]

On Sun, Apr 13, 2014 at 4:25 PM, Chad Austin <ch...@imvu.com> wrote:

Can someone remind me how to run the embind tests within the Emscripten test suite?

python tests/runner.py ALL.test_embind
python tests/runner.py ALL.test_embind_2
python tests/runner.py other.test_embind
 

- Alon

[Chad Austin]

Thanks, running now!

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chad Austin]

To keep everyone updated, I found some time to adjust the pull request, as my previous attempt relied on functionality disallowed by asm.js.

https://github.com/kripken/emscripten/pull/2287

embind/asm.js passes tests and works in my little example programs.

There is a caveat: PRECISE_F32 mode appears to change the function signatures such that "d" becomes "i", which embind doesn't (currently) understand.  Alon, does what I'm seeing make sense to you?  I'm not really sure what to do about that.  It seems that anything that relies on dynCall would break in PRECISE_F32 mode.

embind/asm.js requires fastcomp and will not work with the old asm.js compiler.  If you must use embind and the old compiler, disable asm.js.

Hopefully, this pull request is good enough to merge and embind will be a first class citizen again!

[Alon Zakai]

There should also be 'f' alongside 'd', for float parameters. 'd' converted to 'i' sounds like a bug, got a testcase?

- Alon

[Chad Austin]

On Thu, Apr 24, 2014 at 1:27 PM, Alon Zakai <alon...@gmail.com> wrote:

There should also be 'f' alongside 'd', for float parameters. 'd' converted to 'i' sounds like a bug, got a testcase?

Hm, I've never seen 'f' in any Emscripten signature.  It seems to turn float into 'd'.  All I've ever seen is 'v', 'i', and 'd'.

The PRECISE_F32 case is easy to reproduce in the pull request by simply removing this skip() call:

https://github.com/chadaustin/emscripten/commit/9a530d0e76ea951f761d93ce3aa739b5605acb25#diff-cc4345db19ff44863e9122c74e9f383fR5658

[Chad Austin]

To keep everyone updated, I spoke with Alon on IRC about the PRECISE_F32 case and he showed me what was going on.  In short, the function tables are a little different in the two PRECISE_F32 modes.

I have submitted a pull request for the PRECISE_F32 fix.  https://github.com/kripken/emscripten/pull/2327  (Alon has already merged the previous pull request.)

Once PR 2327 is merged, I believe that embind should work perfectly in fastcomp.  If not, please let me know!

[Joseph Adams]

Hi Chad,

I am interested in getting embind working with fast compile (also new to working with emscripten). How do I get your changes and disable the warnings in emcc?

Thanks

Joe

- Alon

Thanks, running now!

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Chad Austin

Technical Director, IMVU

http://engineering.imvu.com

http://chadaustin.me

--

Chad Austin

Technical Director, IMVU

http://engineering.imvu.com

http://chadaustin.me

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Chad Austin

Technical Director, IMVU

http://engineering.imvu.com

http://chadaustin.me

[Chad Austin]

Hi Joe!

Now that fastcomp and embind/asm.js have been merged from incoming to master, everything should Just Work.

What warnings are you talking about in particular?

Thanks,

Chad

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Joseph Adams]

Hey Chad,

I had warnings coming from emcc saying that the bind flag was not compatible with fast comp. I am not sure if I just had an older version or what, but I commented out the lines throwing the assert and warning and it complies now. Thanks for making bind compatible with fastcomp. 

Joe

--
You received this message because you are subscribed to a topic in the Google Groups "emscripten-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/emscripten-discuss/qFzNcA_9J2Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to emscripten-disc...@googlegroups.com.

[Chad Austin]

Hm, I thought I'd removed all of those.  Are you sure you're on the latest master branch?

[Joseph Adams]

I am probably not. I installed the sdk using the mac installer. I will look into updating it. Thanks for the help!

Joe