Faster emscripten compiler 'fastcomp' almost complete

[Alon Zakai]

Hi,

Work is mostly done on the LLVM backend (codename 'fastcomp') intended to replace the core of emscripten's compiler. This is a proper C++ backend integrated with LLVM, and as such is far faster than the original one which was written in JS. Details about the new compiler are at

https://github.com/kripken/emscripten/wiki/LLVM-Backend

Not all functionality is complete yet in the new compiler, see the list of limitations in that link - stuff like setjmp and C++ exceptions are the main missing things. We'll implement those soon I hope, but I'm sending this email out to see which of the missing features is most important, so I know how to prioritize.

Instructions to build and use the new compiler are in that link as well. Please test when you get a chance, and report any bugs you see (the new compiler passes the emscripten test suite - the parts not using features not present yet - as well as fuzzing, so it seems fairly robust, however like any new compiler bugs are very possible).

- Alon

[Mark Callow]

On 2014/01/07 10:35, Alon Zakai wrote:

Not all functionality is complete yet in the new compiler, see the list of limitations in that link - stuff like setjmp and C++ exceptions are the main missing things. We'll implement those soon I hope, but I'm sending this email out to see which of the missing features is most important, so I know how to prioritize.

Priorities for me would be

1. Linking of asm.js shared modules
2. C++ exceptions
3. SIMD vector types
4. indirectbr
5. legacy GL emulation
6. setjmp/longjmp 
   

Regards

    -Mark

--

注意：この電子メールには、株式会社エイチアイの機密情報が含まれている場合が有ります。正式なメール受信者では無い場合はメール複製、 再配信または情報の使用を固く禁じております。エラー、手違いでこのメールを受け取られましたら削除を行い配信者にご連絡をお願いいたし ます.

NOTE: This electronic mail message may contain confidential and privileged information from HI Corporation. If you are not the intended recipient, any disclosure, photocopying, distribution or use of the contents of the received information is prohibited. If you have received this e-mail in error, please notify the sender immediately and permanently delete this message and all related copies.

[Alon Zakai]

I'm curious why you put linking of modules as #1? One of the reasons for module linking was to avoid long compilation times, which the new compiler solves directly. Or were you using linking for another reason?

- Alon

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Mark Callow]

On 2014/01/07 12:16, Alon Zakai wrote:

I'm curious why you put linking of modules as #1? One of the reasons for module linking was to avoid long compilation times, which the new compiler solves directly. Or were you using linking for another reason?

We have a graphics engine and applications which use it. We would like app developers to be able to work without having the engine source and would like to load the engine code and app code from different .js files. Modules seem like the way to do this but possibly I am misunderstanding. Until now I have been compiling engine & app together and am just about to attempt separating them.

[Chad Austin]

Wow, great!

Our priorities are:

1) C++ exceptions

2) FORCE_ALIGNMENT, SAFE_HEAP etc.

3) SIMD vector types

Regarding GL emulation, I would recommend deleting all of library_gl.js and instead focusing on getting Regal to work reliably.  The Regal project is far more likely to have a complete and usable GL emulation layer.  The built-in emscripten emulation layer doesn't work for anything but the simplest use cases.

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Chad Austin

Technical Director, IMVU

http://www.imvu.com/members/Chad/

[Alon Zakai]

The current modules implementation is not exactly that, it mainly supports static linking, that is, taking two js files and merging them into one, before running that single file. Sounds like you want to load two separate files, which is something that our dlopen support handles. That has some limitations as well, see https://github.com/kripken/emscripten/wiki/Linking

In general, as mentioned in that link, full optimization is only possible when compiling everything all together. But if you don't need absolute maximal performance, either static linking on dlopen can be useful.

- Alon

--

[Alon Zakai]

Thanks for the input. One question, what are you using SAFE_HEAP for specifically?

Agreed about GL emulation in principle, although regal had some limitations as well last I checked, as well as size concerns. Would be good to gather data on the two options to see which wins on speed, compatibility, size, etc.

- Alon

[Chad Austin]

SAFE_HEAP detects accesses through null pointers, right?  Since otherwise null pointer reads and writes are "valid" in asm.js, SAFE_HEAP is useful for catching things early.  (Maybe it should even throw on any accesses below 4096 and above 2 billion or something)

Regal's code size may be a function of its completeness, but it's worth investigating.  I'd even be willing to pay the code size if I could get it to work reliably, but even with -O2 I was running into issues where emscripten was generating excessive numbers of local variables in functions that looked pretty simple.

I think I may have filed a bug on that, but I haven't had time to dig more deeply since before the holidays.

[Alon Zakai]

Yeah, just was curious what you were using SAFE_HEAP for (it does various other stuff as well, like check alignment, check for compiler bugs by checking for impossible types, etc.). The null pointer and alignment checks are certainly things we'll want in the new compiler. Will implement them differently though (see 'design' section in https://github.com/kripken/emscripten/wiki/LLVM-Backend ).

That's btw a nice place to start for people interested to contribute to the new compiler.

- Alon

[Mark Seaborn]

On 7 January 2014 02:31, Mark Callow <callo...@artspark.co.jp> wrote:

On 2014/01/07 10:35, Alon Zakai wrote:

Not all functionality is complete yet in the new compiler, see the list of limitations in that link - stuff like setjmp and C++ exceptions are the main missing things. We'll implement those soon I hope, but I'm sending this email out to see which of the missing features is most important, so I know how to prioritize.

Priorities for me would be

1. Linking of asm.js shared modules
2. C++ exceptions

@Alon:  I haven't looked very closely at how Emscripten implements C++ exceptions, such as how it encodes landingpad clauses in Javascript.  You might find PNaCl's ExceptionInfoWriter.cpp useful for Emscripten.  It's currently used by PNaClSjLjEH.cpp, and it encodes the landingpad clauses into three tables, which are interpreted by libsupc++ or libcxxabi.  From skimming the source, I get the impression that Emscripten is doing something similar.

 

1. SIMD vector types
2. indirectbr

@Alon:  I've just committed a change to PNaCl's LLVM branch which converts indirectbrs to switch instructions (https://gerrit.chromium.org/gerrit/gitweb?p=native_client/pnacl-llvm.git;a=commit;h=a55919f), which you might find useful for Emscripten.

It might not be optimal for Emscripten, though:  if you have to generate a big Javascript switch to handle all the basic blocks (if the relooper fails), then it might be better for blockaddress to return indexes for the big switch, rather than have a second level of switches.

Cheers,

Mark

[Xuejie "Rafael" Xiao]

Hi,

As the maintainer of webruby(https://github.com/xxuejie/webruby), I need setjmp to work in order to port webruby to the new compiler.

However, another thing occurs to me is that with the new compiler, this bug still exists: https://gist.github.com/xxuejie/5574172.

In the old compiler, the workaround of this is to use the old "i386-pc-linux-gnu" target. Since the NaCl team has little interest(https://code.google.com/p/nativeclient/issues/detail?id=2381) of fixing this, my question is: is there a way that we can fix this in the compiler level? Or should I try to fix this in the mruby source code?

Thanks for all the hard work!

Best Regards,

肖雪洁
Xuejie "Rafael" Xiao

[Warren Seine]

On Tuesday, January 7, 2014 2:35:06 AM UTC+1, Alon Zakai wrote:

Not all functionality is complete yet in the new compiler, see the list of limitations in that link - stuff like setjmp and C++ exceptions are the main missing things. We'll implement those soon I hope, but I'm sending this email out to see which of the missing features is most important, so I know how to prioritize.

Exceptions is the only  thing I need right now. 

[Justin Kerk]

On 1/6/2014 5:35 PM, Alon Zakai wrote:
> Not all functionality is complete yet in the new compiler, see the
> list of limitations in that link - stuff like setjmp and C++
> exceptions are the main missing things. We'll implement those soon I
> hope, but I'm sending this email out to see which of the missing
> features is most important, so I know how to prioritize.

For JSMESS:

1. Exceptions
2. setjmp/longjmp (distant second, it's used in a couple places but our
currently targeted system emulations don't need it afaik)

We will want GL of some sort eventually since native MESS supports
shaders and whatnot but currently we're only using the software path
with Emscripten.

[Alon Zakai]

If you have a simple solution in mruby source, that would be best for you I think. If not, then we should see about fixing it in clang. My guess is that it is not a deep problem, just a todo, so likely not hard for us to fix. I've not worked on the clang code before though.

Mark, do you know if pnacl has plans to fix that issue (2381)?

- Alon

On Tue, Jan 7, 2014 at 6:26 PM, Xuejie "Rafael" Xiao <xxu...@gmail.com> wrote:

Hi,

As the maintainer of webruby(https://github.com/xxuejie/webruby), I need setjmp to work in order to port webruby to the new compiler.

However, another thing occurs to me is that with the new compiler, this bug still exists: https://gist.github.com/xxuejie/5574172.

In the old compiler, the workaround of this is to use the old "i386-pc-linux-gnu" target. Since the NaCl team has little interest(https://code.google.com/p/nativeclient/issues/detail?id=2381) of fixing this, my question is: is there a way that we can fix this in the compiler level? Or should I try to fix this in the mruby source code?

Thanks for all the hard work!

Best Regards,

肖雪洁
Xuejie "Rafael" Xiao

On Tue, Jan 7, 2014 at 9:35 AM, Alon Zakai <alon...@gmail.com> wrote:

Hi,

Work is mostly done on the LLVM backend (codename 'fastcomp') intended to replace the core of emscripten's compiler. This is a proper C++ backend integrated with LLVM, and as such is far faster than the original one which was written in JS. Details about the new compiler are at

https://github.com/kripken/emscripten/wiki/LLVM-Backend

Not all functionality is complete yet in the new compiler, see the list of limitations in that link - stuff like setjmp and C++ exceptions are the main missing things. We'll implement those soon I hope, but I'm sending this email out to see which of the missing features is most important, so I know how to prioritize.

Instructions to build and use the new compiler are in that link as well. Please test when you get a chance, and report any bugs you see (the new compiler passes the emscripten test suite - the parts not using features not present yet - as well as fuzzing, so it seems fairly robust, however like any new compiler bugs are very possible).

- Alon

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Alon Zakai]

Note that GL ES 2.0, including GL ES 2.0 emulation, works properly in fastcomp. It is just legacy GL emulation (GL ES 1.x, desktop GL etc.) that does not. Do you know what type of GL JSMESS needs? You mention shaders, so hopefully that is ES 2.0 or something convertable to it.

- Alon

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.

[Alon Zakai]

Thanks for the feedback so far, everyone. Looks like c++ exceptions is the top item, I'll focus on that next.

Meanwhile today I implemented SIMD in fastcomp, so that has been removed from the list of missing features. (SIMD is however not going to be accelerated until browsers JIT that, of course.)

- Alon

[Xuejie "Rafael" Xiao]

Thanks! I'm working on another solution to eliminate the struct type in mruby source by the combination of NaN-boxing and bit manipulations. Hope I can make it into mruby source code, that way we won't have this problem.

BTW: I might be wrong on this, I think the nacl team are using a different approach on this: https://code.google.com/p/nativeclient/issues/detail?id=3338.

Best Regards,

肖雪洁
Xuejie "Rafael" Xiao

[Alon Zakai]

Yes, the PNaCl passes expand out varargs into manual buffer assignments, and the buffer is passed as an argument to the function being called. This is used by fastcomp now. The only issue is the clang frontend errors on a particular type of vararg function, which is separate from this.

- Alon

[Xuejie "Rafael" Xiao]

Oh, okay. Didn't realize this. So I will continue work on the NaN-boxing + bit manipulation method. And hope that works :)

Best Regards,

肖雪洁
Xuejie "Rafael" Xiao

[Justin Kerk]

On 1/8/2014 3:59 PM, Alon Zakai wrote:
> Note that GL ES 2.0, including GL ES 2.0 emulation, works properly in
> fastcomp. It is just legacy GL emulation (GL ES 1.x, desktop GL etc.)
> that does not. Do you know what type of GL JSMESS needs? You mention
> shaders, so hopefully that is ES 2.0 or something convertable to it.

It's not my area of expertise but the current code is presumably regular
ol' desktop GL since there's been no real effort to target mobile
platforms. There are some GLSL shaders but the really fancy shader
effects are currently only available with the HLSL Windows version,
although MooglyGuy has put together a WebGL demo that can hopefully be
integrated at some point: https://www.shadertoy.com/view/ldf3Rf

[Mark Seaborn]

On 8 January 2014 15:56, Alon Zakai <alon...@gmail.com> wrote:

If you have a simple solution in mruby source, that would be best for you I think. If not, then we should see about fixing it in clang. My guess is that it is not a deep problem, just a todo, so likely not hard for us to fix. I've not worked on the clang code before though.

Mark, do you know if pnacl has plans to fix that issue (2381)?

Since you asked, I've prepared a fix and posted it for review:  http://lists.cs.uiuc.edu/pipermail/cfe-commits/Week-of-Mon-20140106/096991.html

Cheers,

Mark

[Mark Seaborn]

On 7 January 2014 15:04, Alon Zakai <alon...@gmail.com> wrote:

On Tue, Jan 7, 2014 at 11:22 AM, Mark Seaborn <msea...@chromium.org> wrote:

On 7 January 2014 02:31, Mark Callow <callo...@artspark.co.jp> wrote:

On 2014/01/07 10:35, Alon Zakai wrote:

Not all functionality is complete yet in the new compiler, see the list of limitations in that link - stuff like setjmp and C++ exceptions are the main missing things. We'll implement those soon I hope, but I'm sending this email out to see which of the missing features is most important, so I know how to prioritize.

Priorities for me would be

1. Linking of asm.js shared modules
2. C++ exceptions

@Alon:  I haven't looked very closely at how Emscripten implements C++ exceptions, such as how it encodes landingpad clauses in Javascript.  You might find PNaCl's ExceptionInfoWriter.cpp useful for Emscripten.  It's currently used by PNaClSjLjEH.cpp, and it encodes the landingpad clauses into three tables, which are interpreted by libsupc++ or libcxxabi.  From skimming the source, I get the impression that Emscripten is doing something similar.

I noticed that, and wasn't sure what it does. How does it relate to LLVM's LowerInvoke pass? That also aims to lower invoke into setjmp AFAICT?

Well, the broader picture is that upstream LLVM has support for SJLJ exception handling, but that support is partly implemented in the backend.  lib/CodeGen/SjLjEHPrepare.cpp is an IR pass which lowers invokes to use setjmp(), but it doesn't do the lowering of the exception info (the catch/filter/cleanup clauses in the landingpad) -- it leaves that to the backend, which lowers landingpads to the .gcc_except_table format.

For PNaCl, we didn't want landingpads' exception info or the .gcc_except_table format to be part of PNaCl's stable ABI, because they're too complex (see https://code.google.com/p/nativeclient/issues/detail?id=3118 for some background).  This means we have to do the lowering of landingpads to data tables as an IR-to-IR pass.  However, there's nothing in upstream LLVM for doing that lowering as an IR-to-IR pass, so I implemented that in ExceptionInfoWriter.cpp.

I haven't actually tried using upstream LLVM's SJLJ EH support, but I assume it works because it's used on iOS (see http://lists.cs.uiuc.edu/pipermail/cfe-dev/2013-October/032656.html).  However, I don't think lib/Transforms/Utils/LowerInvoke.cpp is part of LLVM's currently working SJLJ EH support.  I think it is left over from earlier SJLJ EH support.  I'm not sure it does a useful transformation.  But it definitely doesn't lower landingpads' exception info, so it's not what we want for PNaCl.

 

Currently emscripten lowers invoke into JS try-catch stuff, and uses libcxxabi to do the c++ type stuff, which does sound similar to what you mention there. Still no definite plan how to do it in the new compiler.

Yes, you've told me about the try/catch part before.  What I didn't know was how you're lowering the landingpad clauses.  I've had a closer look now.

It looks like landingpadHandler() in jsifier.js will take a landingpad like this:

  %lp = landingpad ... personality ...

      catch i8* @exception_type1

      catch i8* @exception_type2

and convert it to:

   __cxa_find_matching_catch(-1, -1, exception_type1, exception_type2)

where __cxa_find_matching_catch() is implemented in library.js and uses "arguments" to read the variable-arguments list.

So it look like this doesn't handle "filter" clauses (for C++ exception specifications).  Presumably you ignore the "cleanup" clause, but it's OK to treat every landingpad as if it has a "cleanup" clause.  (In order for the IR to be inliner-friendly, Clang generates landingpad blocks which can handle being entered even if the exception being thrown didn't match a "catch" clause.)

So you're encoding the landingpad's exception info as a list of arguments in Javascript.  We could have done something similar in PNaCl, but I figured it was a bit more efficient to generate a read-only table in the data segment (as C++ exception handling normally does) and pass a reference to that.

Cheers,

Mark

[Xuejie "Rafael" Xiao]

Mark: Thanks a lot for the patch!

I did some test using the patch. Clang would indeed stop issuing errors, however, we are still hitting some errors in llvm. A detailed log containing outputs before and after the patch can be found at here: https://gist.github.com/xxuejie/8384185.

So is this because of some restrictions in emscripten-fastcomp?

Best Regards,

肖雪洁
Xuejie "Rafael" Xiao

[Mark Seaborn]

On 12 January 2014 04:50, Xuejie "Rafael" Xiao <xxu...@gmail.com> wrote:

Mark: Thanks a lot for the patch!

I did some test using the patch. Clang would indeed stop issuing errors, however, we are still hitting some errors in llvm. A detailed log containing outputs before and after the patch can be found at here: https://gist.github.com/xxuejie/8384185.

So is this because of some restrictions in emscripten-fastcomp?

Good point.  I didn't notice that because I didn't test it end-to-end.  Here's a fix for the IR simplification passes in the LLVM branch: https://codereview.chromium.org/135953002

Cheers,

Mark

[Xuejie "Rafael" Xiao]

Thanks Mark! Now we don't have the LLVM error as before.

However, I think this time we are hitting something in emscripten. The same gist is updated: https://gist.github.com/xxuejie/8384185. This time we are getting an error in JSBackend. So Alon, can you help take a look at this?

Thanks all of you for your help!

Best Regards,

肖雪洁
Xuejie "Rafael" Xiao

[Alon Zakai]

Sure, but any chance you can send me the bitcode that the backend fails on? EMCC_DEBUG=1, and grab emcc-1-linktime.bc from /tmp/emscripten_temp.

- Alon

[Alon Zakai]

Ok, C++ exceptions support is now working in fastcomp. If your project was waiting on that to try fastcomp, now is the time to test and report issues :)

Next on my list is setjmp/longjmp. For this I plan to try a more optimized approach than we currently have. Right now the original compiler will not reloop functions using setjmp, which means if you have a tight loop in such a function it will be slow. Instead, I think we can do a nearly-complete relooping, by adding branches where a longjmp could return to the setjmp, and adding a feature to the relooper that is sort of like LLVM's indirect branch instruction. Thinking about that now, i might want to do LLVM indirectbr first and implement setjmp using that.

- Alon

[Xuejie "Rafael" Xiao]

Definitely! Here's the file: https://copy.com/xGKJsUSI93Fm.

Let me know if you need more information here. Thanks for the help!

Best Regards,

肖雪洁
Xuejie "Rafael" Xiao

[Alon Zakai]

That bitcode file is only 2k in size and compiles ok in fastcomp, perhaps something got mixed up?

- Alon

[Alon Zakai]

Oh ok, it is meant to be run just through llc, then it fails.

Looks like the issue is that the pnacl passes leave a type in there,

%union.anon = type { float }

So the problem occurred before this step, and the emcc-0 file would be useful.

- Alon

[Mark Seaborn]

On 13 January 2014 20:56, Alon Zakai <alon...@gmail.com> wrote:

Oh ok, it is meant to be run just through llc, then it fails.

Looks like the issue is that the pnacl passes leave a type in there,

%union.anon = type { float }

So the problem occurred before this step, and the emcc-0 file would be useful.

I see the problem.  It's that ExpandStructRegs doesn't handle nested struct types.  vatest.c at https://gist.github.com/xxuejie/8384185 reproduces the bug.

I'll prepare a fix.  Thanks for the bug report!

Mark

[Xuejie "Rafael" Xiao]

Thank both of you for the help! In case if you need it, here's the emcc-0 file: https://copy.com/6gak7Rf7aRil

Best Regards,

肖雪洁
Xuejie "Rafael" Xiao

[Alon Zakai]

Setjmp/longjmp now works in fastcomp.

I ended up doing this in a way that does not use indirectbr. I also took a shortcut in that C++ exceptions will not work together with setjmp, that is fixable but i'm not sure how important to prioritize that - please let me know if you have a codebase using both, and in what way.

- Alon

[Justin Kerk]

On 1/20/2014 4:08 PM, Alon Zakai wrote:
> I also took a shortcut in that C++ exceptions will not work together
> with setjmp, that is fixable but i'm not sure how important to
> prioritize that - please let me know if you have a codebase using
> both, and in what way.

Compiling JSMESS bombs out with the error that you can't use both. AFAIK
they are in different areas of the codebase (e.g. the MAME device system
uses exceptions, but we also incorporate libflac which uses setjmp).

[Alon Zakai]

Yeah, I guess large projects like JSMESS might run into that. Ok, I pushed some fixes that allow this case, let me know if that resolves the issue for you.

I have not checked if "crossing" c++ exceptions in flight with longjmps similarly in flight would work - but come to think of it, I'm not sure that ever worked in the original compiler either.

- Alon

[Justin Kerk]

On 1/21/2014 2:02 PM, Alon Zakai wrote:
> Yeah, I guess large projects like JSMESS might run into that. Ok, I
> pushed some fixes that allow this case, let me know if that resolves
> the issue for you.

It compiles again now, but still abort()s early in execution. I don't
have time to investigate right now but the number of unresolved symbols
is a bit worrisome:

Warning: Variable __init_array_start not referenced
Warning: Variable __init_array_end not referenced
Warning: Variable __fini_array_start not referenced
Warning: Variable __fini_array_end not referenced
warning: unresolved symbol: UItoF
warning: unresolved symbol: UItoD
warning: Legacy GL function (glVertexPointer) called. You need to
compile with -s LEGACY_GL_EMULATION=1 to enable legacy GL emulation.
warning: Legacy GL function (glGenVertexArrays) called. You need to
compile with -s LEGACY_GL_EMULATION=1 to enable legacy GL emulation.
warning: Legacy GL function (glMatrixMode) called. You need to compile
with -s LEGACY_GL_EMULATION=1 to enable legacy GL emulation.
warning: Legacy GL function (glBegin) called. You need to compile with
-s LEGACY_GL_EMULATION=1 to enable legacy GL emulation.
warning: Legacy GL function (glLoadIdentity) called. You need to compile
with -s LEGACY_GL_EMULATION=1 to enable legacy GL emulation.
warning: unresolved symbol: SDL_CreateYUVOverlay
warning: unresolved symbol: BDtoILow
warning: unresolved symbol: emscripten_get_longjmp_result
warning: unresolved symbol: pthread_attr_setdetachstate
warning: unresolved symbol: emscripten_landingpad
warning: unresolved symbol: emscripten_prep_setjmp
warning: unresolved symbol: SItoD
warning: unresolved symbol: SItoF
warning: unresolved symbol: FPtoIHigh
warning: unresolved symbol: emscripten_setjmp
warning: unresolved symbol: _saveSetjmp
warning: unresolved symbol: emscripten_resume
warning: unresolved symbol: emscripten_postinvoke
warning: unresolved symbol: emscripten_check_longjmp
warning: unresolved symbol: emscripten_preinvoke
warning: unresolved symbol: SDL_GL_LoadLibrary
warning: unresolved symbol: pthread_create
warning: unresolved symbol: SDL_UnlockYUVOverlay
warning: unresolved symbol: BDtoIHigh
warning: unresolved symbol: BItoD
warning: unresolved symbol: SDL_LockYUVOverlay
warning: unresolved symbol: FPtoILow
warning: unresolved symbol: SDL_DisplayYUVOverlay
warning: unresolved symbol: SDL_FreeYUVOverlay
warning: unresolved symbol: _ZTISt9exception
warning: unresolved symbol: __fsmu8
warning: Output contains some very large functions, consider using
OUTLINING_LIMIT to break them up (see settings.js)

Some of the SDL ones occur without using fastcomp and don't seem to be
relevant for how we're running but I wonder about the emscripten_ ones.

[Alon Zakai]

You can ignore the emscripten_ ones, also several others like BItoD, those are false positives. I should figure out how to clean those out.

- Alon

[wolfviking0]

Hi,

I just rebuild Unigine using fast comp, I have a warning about the llvm version (LLVM version appears incorrect (seeing "clang version 3.3 (https://chromium.googlesource.com/native_client/pnacl-clang 0196efefb27e9a768ae4d50c7ff75fb3895d4617) (https://github.com/kripken/emscripten-fastcomp 572bba39dcadf613afa586320d83aa3fc48a3cf8)", expected "clang version 3.2")

I am not sure if is an important warning but I have an error during the build.

Tony

** CONSOLE **

JAVA_HEAP_SIZE=8096m EMCC_FAST_COMPILER=1 EMCC_DEBUG=1 /Users/aliot/Emscripten/emcc /Users/aliot/Actisku/jsbuild/lib/libfreetype.o /Users/aliot/Actisku/jsbuild/lib/libjpeg.o /Users/aliot/Actisku/jsbuild/lib/libogg.o /Users/aliot/Actisku/jsbuild/lib/libpng.o /Users/aliot/Actisku/jsbuild/lib/libregexpr.o /Users/aliot/Actisku/jsbuild/lib/libunzip.o /Users/aliot/Actisku/jsbuild/lib/libz.o  /Users/aliot/Actisku/jsbuild/lib/libengine.o -O2 -s WARN_ON_UNDEFINED_SYMBOLS=1 -s ASM_JS=1 -s FULL_ES2=1  \

-o /Users/aliot/Actisku/jsbuild/html/libUnigine.js

DEBUG:root:emscript: llvm backend

DEBUG:root:  ..1..

Warning: Variable __init_array_start not referenced

Warning: Variable __init_array_end not referenced

Warning: Variable __fini_array_start not referenced

Warning: Variable __fini_array_end not referenced

/Users/aliot/Desktop/webcl/compilo/llvm/fastcomp/bin/llvm-dis: Could not open temp1.bc: No such file or directory

DEBUG:root:  ..3..

Call parameter type does not match function signature!

  %33 = load float* %.bc11, align 4

 double  %34 = call i32 @FPtoILow(float %33)

Call parameter type does not match function signature!

  %33 = load float* %.bc11, align 4

 double  %35 = call i32 @FPtoIHigh(float %33)

Broken module found, compilation aborted!

0  opt                      0x0000000101d53b58 void* llvm::object_creator<std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >() + 12424

1  opt                      0x0000000101d54044 void* llvm::object_creator<std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >() + 13684

2  libsystem_platform.dylib 0x00007fff96f5f5aa _sigtramp + 26

3  opt                      0x0000000101c6107e std::__1::vector<std::__1::pair<llvm::AttributeSet, unsigned int>, std::__1::allocator<std::__1::pair<llvm::AttributeSet, unsigned int> > >::__append(unsigned long) + 11710

4  opt                      0x0000000101d53e66 void* llvm::object_creator<std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >() + 13206

5  opt                      0x0000000101d0f65d void std::__1::vector<llvm::StructType*, std::__1::allocator<llvm::StructType*> >::__push_back_slow_path<llvm::StructType* const>(llvm::StructType* const&) + 28685

6  opt                      0x0000000101d0f427 void std::__1::vector<llvm::StructType*, std::__1::allocator<llvm::StructType*> >::__push_back_slow_path<llvm::StructType* const>(llvm::StructType* const&) + 28119

7  opt                      0x0000000101cfb29c std::__1::vector<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord> >::__swap_out_circular_buffer(std::__1::__split_buffer<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord>&>&) + 27948

8  opt                      0x0000000101cfb48b std::__1::vector<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord> >::__swap_out_circular_buffer(std::__1::__split_buffer<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord>&>&) + 28443

9  opt                      0x0000000101cfb752 std::__1::vector<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord> >::__swap_out_circular_buffer(std::__1::__split_buffer<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord>&>&) + 29154

10 opt                      0x0000000101cfbfbd std::__1::vector<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord> >::__swap_out_circular_buffer(std::__1::__split_buffer<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord>&>&) + 31309

11 opt                      0x0000000101cfc35d std::__1::vector<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord> >::__swap_out_circular_buffer(std::__1::__split_buffer<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord>&>&) + 32237

12 opt                      0x00000001015915f7 void std::__1::vector<std::__1::pair<llvm::BasicBlock*, llvm::SuccIterator<llvm::TerminatorInst*, llvm::BasicBlock> >, std::__1::allocator<std::__1::pair<llvm::BasicBlock*, llvm::SuccIterator<llvm::TerminatorInst*, llvm::BasicBlock> > > >::__push_back_slow_path<std::__1::pair<llvm::BasicBlock*, llvm::SuccIterator<llvm::TerminatorInst*, llvm::BasicBlock> > const>(std::__1::pair<llvm::BasicBlock*, llvm::SuccIterator<llvm::TerminatorInst*, llvm::BasicBlock> > const&) + 7927

13 libdyld.dylib            0x00007fff9878c5fd start + 1

14 libdyld.dylib            0x0000000000000005 start + 1736915465

Stack dump:

0. Program arguments: /Users/aliot/Desktop/webcl/compilo/llvm/fastcomp/bin/opt /tmp/emscripten_temp/tmpjCxa9g.1.bc -pnacl-abi-simplify-postopt -o /tmp/emscripten_temp/tmpPHrGPa.3.bc 

1. Running pass 'Function Pass Manager' on module '/tmp/emscripten_temp/tmpjCxa9g.1.bc'.

2. Running pass 'Module Verifier' on function '@_ZNK8Variable11getLongSafeEv'

Traceback (most recent call last):

  File "/Users/aliot/Emscripten/emscripten.py", line 1337, in <module>

    _main(environ=os.environ)

  File "/Users/aliot/Emscripten/emscripten.py", line 1325, in _main

    temp_files.run_and_clean(lambda: main(

  File "/Users/aliot/Emscripten/tools/tempfiles.py", line 39, in run_and_clean

    return func()

  File "/Users/aliot/Emscripten/emscripten.py", line 1333, in <lambda>

    DEBUG_CACHE=DEBUG_CACHE,

  File "/Users/aliot/Emscripten/emscripten.py", line 1220, in main

    jcache=jcache, temp_files=temp_files, DEBUG=DEBUG, DEBUG_CACHE=DEBUG_CACHE)

  File "/Users/aliot/Emscripten/emscripten.py", line 764, in emscript_fast

    assert os.path.exists(temp3)

AssertionError

Traceback (most recent call last):

  File "/Users/aliot/Emscripten/emcc", line 1691, in <module>

    final = shared.Building.emscripten(final, append_ext=False, extra_args=extra_args)

  File "/Users/aliot/Emscripten/tools/shared.py", line 1221, in emscripten

    assert os.path.exists(filename + '.o.js') and len(open(filename + '.o.js', 'r').read()) > 0, 'Emscripten failed to generate .js: ' + str(compiler_output)

AssertionError: Emscripten failed to generate .js: 

make: *** [all_engine_js] Error 1

[Alon Zakai]

The warning can be ignored, but the rest of the output looks like it is old - are you on very latest emscripten incoming? (and also latest fastcomp?)

- Alon

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

[wolfviking0]

Hi Alon I just update everything, and the error seems different now.

I just update you last incoming commit (avoid unresolved symbol warnings on misc funcs in fast comp) and you last fastcomp commit (typo).

Now the error I have is :

Warning: Variable __init_array_start not referenced

Warning: Variable __init_array_end not referenced

Warning: Variable __fini_array_start not referenced

Warning: Variable __fini_array_end not referenced

Call parameter type does not match function signature!

  %33 = load float* %.bc11, align 4

 double  %34 = call i32 @FPtoILow(float %33)

Call parameter type does not match function signature!

  %33 = load float* %.bc11, align 4

 double  %35 = call i32 @FPtoIHigh(float %33)

Broken module found, compilation aborted!

0  opt                      0x0000000100f0db58 void* llvm::object_creator<std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >() + 12424

1  opt                      0x0000000100f0e044 void* llvm::object_creator<std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >() + 13684

2  libsystem_platform.dylib 0x00007fff84f9c5aa _sigtramp + 26

3  opt                      0x0000000100e1b07e std::__1::vector<std::__1::pair<llvm::AttributeSet, unsigned int>, std::__1::allocator<std::__1::pair<llvm::AttributeSet, unsigned int> > >::__append(unsigned long) + 11710

4  opt                      0x0000000100f0de66 void* llvm::object_creator<std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > >() + 13206

5  opt                      0x0000000100ec965d void std::__1::vector<llvm::StructType*, std::__1::allocator<llvm::StructType*> >::__push_back_slow_path<llvm::StructType* const>(llvm::StructType* const&) + 28685

6  opt                      0x0000000100ec9427 void std::__1::vector<llvm::StructType*, std::__1::allocator<llvm::StructType*> >::__push_back_slow_path<llvm::StructType* const>(llvm::StructType* const&) + 28119

7  opt                      0x0000000100eb529c std::__1::vector<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord> >::__swap_out_circular_buffer(std::__1::__split_buffer<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord>&>&) + 27948

8  opt                      0x0000000100eb548b std::__1::vector<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord> >::__swap_out_circular_buffer(std::__1::__split_buffer<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord>&>&) + 28443

9  opt                      0x0000000100eb5752 std::__1::vector<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord> >::__swap_out_circular_buffer(std::__1::__split_buffer<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord>&>&) + 29154

10 opt                      0x0000000100eb5fbd std::__1::vector<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord> >::__swap_out_circular_buffer(std::__1::__split_buffer<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord>&>&) + 31309

11 opt                      0x0000000100eb635d std::__1::vector<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord> >::__swap_out_circular_buffer(std::__1::__split_buffer<llvm::Module::NeededRecord, std::__1::allocator<llvm::Module::NeededRecord>&>&) + 32237

12 opt                      0x000000010074b5f7 void std::__1::vector<std::__1::pair<llvm::BasicBlock*, llvm::SuccIterator<llvm::TerminatorInst*, llvm::BasicBlock> >, std::__1::allocator<std::__1::pair<llvm::BasicBlock*, llvm::SuccIterator<llvm::TerminatorInst*, llvm::BasicBlock> > > >::__push_back_slow_path<std::__1::pair<llvm::BasicBlock*, llvm::SuccIterator<llvm::TerminatorInst*, llvm::BasicBlock> > const>(std::__1::pair<llvm::BasicBlock*, llvm::SuccIterator<llvm::TerminatorInst*, llvm::BasicBlock> > const&) + 7927

13 libdyld.dylib            0x00007fff8b22f5fd start + 1

14 libdyld.dylib            0x000000000000000a start + 1960643086

Stack dump:

0. Program arguments: /Volumes/APPLE_MEDIA/WORKSPACE/compilo/llvm/fastcomp/bin/opt /tmp/tmpWsFkjp/libUnigine.bc -strip-debug -internalize -internalize-public-api-list=main,malloc,free -globaldce -pnacl-abi-simplify-preopt -pnacl-abi-simplify-postopt -o /tmp/tmpWsFkjp/libUnigine.bc.opt.bc 

1. Running pass 'Function Pass Manager' on module '/tmp/tmpWsFkjp/libUnigine.bc'.

2. Running pass 'Module Verifier' on function '@_ZNK8Variable11getLongSafeEv'

Traceback (most recent call last):

  File "/Volumes/APPLE_MEDIA/WORKSPACE/webgl/emscripten/emcc", line 1822, in <module>

    shared.Building.llvm_opt(final, link_opts)

  File "/Volumes/APPLE_MEDIA/WORKSPACE/webgl/emscripten/tools/shared.py", line 1173, in llvm_opt

    assert os.path.exists(target), 'Failed to run llvm optimizations: ' + output

AssertionError: Failed to run llvm optimizations: 

make: *** [all_engine_js] Error 1

Mac-mini-de-Anthony:unigine_work al$  

[Alon Zakai]

Thanks, I see we were missing fptoui|si on floats, we had only doubles. I added that now.

- Alon

--

[asnapper]

Hi

I have a fairly large codebase (game engine using gles2) that is already working pretty well using the old compiler backend. As building takes pretty long (~5minutes), i'd like to move on the the llvm backend as soon as possible.

Using the new backend, i get a few warnings while building:

Warning: Variable __init_array_start not referenced
Warning: Variable __init_array_end not referenced
Warning: Variable __fini_array_start not referenced
Warning: Variable __fini_array_end not referenced

warning: unresolved symbol: _ZTISt9exception

warning: unresolved symbol: glMapBuffer

warning: unresolved symbol: glUnmapBuffer 

I'm using exactly the same options as with the old backend

-s TOTAL_MEMORY=536870912 \

-s WARN_ON_UNDEFINED_SYMBOLS=1 \

-s CASE_INSENSITIVE_FS=1

As I have read here, the unresolved symbols could very well just be false positives. could you confirm this?

Furthermore, at runtime, i get the following error:

uncaught exception: abort() at stackTrace@http://localhost/editor.js:12346

abort@http://localhost/editor.js:1933500

_abort@http://localhost/editor.js:19932

invoke_iii@http://localhost/editor.js:20865

invoke_vii@http://localhost/editor.js:20379

invoke_vi@http://localhost/editor.js:20622

invoke_vi@http://localhost/editor.js:20622

invoke_vi@http://localhost/editor.js:20622

invoke_vii@http://localhost/editor.js:20379

callMain@http://localhost/editor.js:1933397

doRun@http://localhost/editor.js:1933450

run/<@http://localhost/editor.js:1933462

I cannot relate to what in my codebase triggers this error, so i'm wondering if I am hitting a known limitation of the new backend or if this is something you may want to look into?

Thanks

Matt

[asnapper]

On Thursday, January 30, 2014 11:59:08 AM UTC+1, asnapper wrote:

I cannot relate to what in my codebase triggers this error, so i'm wondering if I am hitting a known limitation of the new backend or if this is something you may want to look into?

Just a little follow-up, I did some good old "printf debugging", and found the code it fails on in my codebase. For some reason fastcomp chokes on our sort algorithm:

template <typename T>

inline void Swap(T& a, T& b)

{

T temp = a;

a = b;

b = temp; // here it fails

}

[Alon Zakai]

Those warnings can be ignored, assuming you do not need to call typeinfo(std::exception), glMapBuffer or glUnmapBuffer.

The runtime abort is not a known limitation, that is something we need to figure out and solve. Any chance you can create a standalone testcase using that Swap function that I can debug? (I tried using it to swap some ints and that was fine, so it must be something more complex.)

- Alon

--

[Mark Callow]

On 2014/01/30 3:59, asnapper wrote:

Hi

I have a fairly large codebase (game engine using gles2) that is already working pretty well using the old compiler backend. As building takes pretty long (~5minutes), i'd like to move on the the llvm backend as soon as possible.

Using the new backend, i get a few warnings while building:

Warning: Variable __init_array_start not referenced
Warning: Variable __init_array_end not referenced
Warning: Variable __fini_array_start not referenced
Warning: Variable __fini_array_end not referenced
warning: unresolved symbol: _ZTISt9exception

warning: unresolved symbol: glMapBuffer

warning: unresolved symbol: glUnmapBuffer 

{Map,Unmap}Buffer are not part of unextended OpenGL ES 2. I don't know whose "gles2" implementation you were using previously but your code base appears to be inadvertently using some extension that provides these functions.

Regards

    -Mark

--

注意：この電子メールには、株式会社エイチアイの機密情報が含まれている場合が有ります。正式なメール受信者では無い場合はメール複製、 再配信または情報の使用を固く禁じております。エラー、手違いでこのメールを受け取られましたら削除を行い配信者にご連絡をお願いいたし ます.

NOTE: This electronic mail message may contain confidential and privileged information from HI Corporation. If you are not the intended recipient, any disclosure, photocopying, distribution or use of the contents of the received information is prohibited. If you have received this e-mail in error, please notify the sender immediately and permanently delete this message and all related copies.

[asnapper]

{Map,Unmap}Buffer are not part of unextended OpenGL ES 2. I don't know whose "gles2" implementation you were using previously but your code base appears to be inadvertently using some extension that provides these functions.

As it is a multiplatform/multirenderer(directx, opengl, gles2 and trough emscripten webgl) engine, there are several different codepaths for each graphics backend, and in the case of webgl, it does not depend glMapBuffer/OES.

Maybe the 'old' compiler did a better job when it comes to dead code elimination?

[asnapper]

The runtime abort is not a known limitation, that is something we need to figure out and solve. Any chance you can create a standalone testcase using that Swap function that I can debug? (I tried using it to swap some ints and that was fine, so it must be something more complex.)

Yeah i tried that too, even added our whole container template library to the testcase, but i cannot reproduce the error. The best i could offer would be a --save-bc output (if my employer is ok with that)

Btw: no lecture from you at mloc.js ? ;-)

[Alon Zakai]

Hmm, dead code elimination is the same, except for perhaps llvm changes from 3.2 to 3.3 (fastcomp is 3.3-based). I would expect in both that if you check at runtime whether to use those functions, you would see a warning about the symbol not being defined, because it can't statically decide it is not used.

- Alon

--

[Alon Zakai]

Sure, if you can send a bc, i can take a look at that.

No mloc.js for me this year, I have other commitments - the timing is not good because it's just a short time before GDC...

- Alon

--