How to track a potential Emscripten bug?!

[Christoph Husse]

Hi,

I am currently trying to convert my so far biggest C++ project to javascript. I have some experience with Emscripten by now but thise here gives me headaches. I tried for one day, but no success.

The problem is as follows:

I have a library containing a few megabytes of C++ code (so pretty big). It compiles fine on Visual Studio 2013, Intel Composer 14, GCC4.8, Clang 3.3... All can run it with assertion, valgrind does not notify any issue, Intel Memory Analyzer does not report one either.

Now when I compile it to Javascript with Emscripten the weirdest things happen. It plays sound, but the graphics is totally screwed up and the game just doesn't work. I tried with all optimization options, debug info, SAFE_HEAP, typed arrays settings, alignment, function pointer aliasing, etc... No success. No error reported, no exception thrown, it just doesn't work. Same for any browser too, so not a Chrome issue.

NOW, my last resort is "Autodebug", which proposedly should be able to track down such issues (personally i believe its an Emscripten bug, since everythign else works flawlessly). But I can't get it to work. When I generate the Bitcode file with emscripten, how can I possibly run this file with a normal CPU?

I tried "lli", "llc" and so on. "lli" even crashes on the most simple applications (like a main() routine that fills a std::string vector with the apssed arguments, what gives?) with an error like this:

lli: /home/natalie/Documents/llvm/lib/ExecutionEngine/JIT/JIT.cpp:622: void llvm::JIT::runJITOnFunctionUnlocked(llvm::Function*, const llvm::MutexGuard&): Assertion `!isAlreadyCodeGenerating && "Error: Recursive compilation detected!"' failed.

0  lli             0x0000000000d1a9ef

1  lli             0x0000000000d1aee9

2  libpthread.so.0 0x00007fc73cc37030

3  libc.so.6       0x00007fc73bf2c475 gsignal + 53

4  libc.so.6       0x00007fc73bf2f6f0 abort + 384

5  libc.so.6       0x00007fc73bf25621 __assert_fail + 241

6  lli             0x000000000085168c

7  lli             0x0000000000851799 llvm::JIT::getPointerToFunction(llvm::Function*) + 265

8  lli             0x000000000085a13e

9  lli             0x000000000085bc4b

10 lli             0x000000000058cb21

11 lli             0x0000000000cb44df llvm::FPPassManager::runOnFunction(llvm::Function&) + 607

12 lli             0x0000000000cb7a1b llvm::FunctionPassManagerImpl::run(llvm::Function&) + 139

13 lli             0x0000000000cb7b99 llvm::FunctionPassManager::run(llvm::Function&) + 105

14 lli             0x0000000000850f77 llvm::JIT::jitTheFunction(llvm::Function*, llvm::MutexGuard const&) + 39

15 lli             0x00000000008515dc llvm::JIT::runJITOnFunctionUnlocked(llvm::Function*, llvm::MutexGuard const&) + 28

16 lli             0x0000000000851799 llvm::JIT::getPointerToFunction(llvm::Function*) + 265

17 lli             0x000000000085a6b6

18 lli             0x0000000000625fdd

19 lli             0x0000000000625afa X86CompilationCallback + 74

Stack dump:

0. Program arguments: /home/natalie/Documents/llvm/build/Release+Asserts/bin/lli ./SNES.bc 

1. Running pass 'X86 Machine Code Emitter' on function '@_ZNSt3__18ios_base4InitC2Ev'

Like I always get the "recusive compilation" bit here...

With "llc" I always get "inavlid instruction" or "Target doesnt support this output file".

Now I have somewhat reached the end. I have no idea what else to try.

So please tell me, how do I get "autodebug" to work on a main() routine with an "std::vector" for starters? because right now this seems like a lost cause.

Thanks :)

[Jukka Jylänki]

I don't know about autodebug, but since you say that the graphics are screwed up, I would start reducing this down to the first draw command that fails, and then trying to extract that as a standalone test case that's small enough to showcase a bug. If it's something else than the result of a draw command, then if other options fail, tracing the program execution with prints and bisecting to a place where the native and JS versions differ, would be a good starting point. That should give more information than "the game just doesn't work".

For issues with llc and lli: neither of those two tools are documented in Emscripten, and they are intended only to be used internally by the backend. It is possible that if you don't drive those tools exactly like emcc does during its operation, that it can fail. You can try posting your command line and input files to get more guidance there.

Best,

   Jukka

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Christoph Husse]

Well I didn't mention it but no, the graphics are not the issue. Everything is rendered into a texture and it works exactly the same as in other games I wrote. Its hard to describe, but there is stuff rendered but its like "mirrored" on a tile basis. Basically that means somewhere a sign flips or I don't know (the game renders the screen on a per pixel basis internally, since its an emulator; the SDL part is rendering one screen-quad texture, not much that can go wrong here). Also the game doesn't work means that nothing actually happens. It stays at the same frame all the time, just sound is playing.

Anyway, its the core game library that is the issue here. And it doesn't contain anything of SDL/audio or whatever, its just C++ code without anything except standard headers.

About the "recursive compilation" bug. It seems to be a now fixed LLVM bug but maybe emscripten has older sources. The "assertion failed" is misleading as the flag that was set there shouldn't be set. So basically it reports an error that doesn't exist. Anyway, Emscripten offers Autodebug, which seems to be an incredibly important tool for debugging issues. So its definitely not an option to say its not supported. This needs to work! There are not many choices of tracking down compiler bugs in large applications otherwise.

When I get it to work I may write up a "howto" for this thing. 

[Jukka Jylänki]

I am not saying that autodebug would not be supported, just that I do not know how to operate it, and was skipping that to refer to the other parts in your mail.

Emscripten Clang backend is currently based on top of Clang 3.3, so it is not the latest version (Clang 3.4 is the latest). If the fix is simple, perhaps it could be backported?

--

[Christoph Husse]

Well the fix is building LLVM without assertions :).

But that doesn't get me any further. Now I get "__dso_handle" undefined, when I run "lli"...

Seems that this is some vodoo magic thing and defining it myself with:

extern "C"

{

const void *const __dso_handle = &__dso_handle;

}

Doesn't do a damn thing :O.

Also I get segmentation faults if I add more code. This is getting really weird.

Can someone who knows the autodebug feature please tell me what's going on here :). A simple main program like this:

int main(int argc, char** argv)

{

std::vector<std::string> args;

for (int i = 0; i < argc; i++)

{

args.emplace_back(argv[i]);

}

}

Will completely destroy LLVM in its current emscripten github checkout :)... So it shouldn't be hard to track down the issue.

I kinda have the feeling there is more to it than just running "lli" on emscripten bitcode because it just seems so completely broken that I can't believe this would have passed tests.

[Christoph Husse]

Ok here some step-by-step testcase:

THIS IS "main.cpp":

#include <vector>

#include <string>

#include <iostream>

int main(int argc, char** argv)

{

std::vector<std::string> args;

for (int i = 0; i < argc; i++)

{

args.emplace_back(argv[i]);

}

std::cout << "Hello World!" << std::endl;

return 0;

}

NOW COMPILE

/home/natalie/Documents/emscripten/em++ -std=c++11 main.cpp -o main.js

/home/natalie/Documents/emscripten/em++ -std=c++11 main.cpp -o main.ll

/home/natalie/Documents/fastcomp/build/Release/bin/llvm-link ./main.ll ~/.emscripten_cache/libcxx.bc -o main.bc

/home/natalie/Documents/fastcomp/build/Release/bin/lli ./main.bc

RESULT:

Node-JS does on "main.js" what it should...

LLI crashes with:

/home/natalie/Documents/fastcomp/build/Release/bin/lli ./main.bc

0  lli             0x0000000000ce5992 llvm::sys::PrintStackTrace(_IO_FILE*) + 34

1  lli             0x0000000000ce5619

2  libpthread.so.0 0x00007fe98441a030

3  lli             0x00000000008a5840 llvm::MachineJumpTableInfo::getEntrySize(llvm::DataLayout const&) const + 0

4  lli             0x000000000084d1c3

5  lli             0x0000000000854d53

6  lli             0x000000000059b60f

7  lli             0x0000000000c85fff llvm::FPPassManager::runOnFunction(llvm::Function&) + 607

8  lli             0x0000000000c860dc llvm::FunctionPassManagerImpl::run(llvm::Function&) + 124

9  lli             0x0000000000c861cd llvm::FunctionPassManager::run(llvm::Function&) + 109

10 lli             0x00000000008468f7 llvm::JIT::jitTheFunction(llvm::Function*, llvm::MutexGuard const&) + 39

11 lli             0x0000000000846eef llvm::JIT::runJITOnFunctionUnlocked(llvm::Function*, llvm::MutexGuard const&) + 15

12 lli             0x0000000000847049 llvm::JIT::getPointerToFunction(llvm::Function*) + 265

13 lli             0x0000000000855cef

14 lli             0x000000000059b60f

15 lli             0x0000000000c85fff llvm::FPPassManager::runOnFunction(llvm::Function&) + 607

16 lli             0x0000000000c860dc llvm::FunctionPassManagerImpl::run(llvm::Function&) + 124

17 lli             0x0000000000c861cd llvm::FunctionPassManager::run(llvm::Function&) + 109

18 lli             0x00000000008468f7 llvm::JIT::jitTheFunction(llvm::Function*, llvm::MutexGuard const&) + 39

19 lli             0x0000000000846eef llvm::JIT::runJITOnFunctionUnlocked(llvm::Function*, llvm::MutexGuard const&) + 15

20 lli             0x0000000000847049 llvm::JIT::getPointerToFunction(llvm::Function*) + 265

21 lli             0x0000000000854aee

22 lli             0x000000000062ef4f

23 lli             0x000000000062eb2a X86CompilationCallback + 74

Stack dump:

0. Program arguments: /home/natalie/Documents/fastcomp/build/Release/bin/lli ./main.bc 

1. Running pass 'X86 Machine Code Emitter' on function '@_ZNSt3__18ios_base4InitC2Ev'

2. Running pass 'X86 Machine Code Emitter' on function '@_ZNSt3__16localeC2Ev'

Segmentation fault

[Bruce Mitchener]

On Tue, May 6, 2014 at 5:49 PM, Christoph Husse <thesai...@googlemail.com> wrote:

I kinda have the feeling there is more to it than just running "lli" on emscripten bitcode because it just seems so completely broken that I can't believe this would have passed tests.

The autodebug that I remember was for LLVM rather than emscripten (bugpoint).  You're probably going to have to wait for Alon to wake up and respond to emails for further information.

It may well be that lli doesn't work with emscripten's bitcode any longer after the change to the current target triple.

Good luck!

 - Bruce

[Christoph Husse]

It may well be that lli doesn't work with emscripten's bitcode any longer after the change to the current target triple.

Well that would be really bad. How else would I be able to find the offending machine instruction that might be translated the wrong way by Emscripten in millions of statements?

Also I think its  a C++ issue. I have now started to replace things like std::vector<> & stuff with my own ultra slim & simple implementations which basically all come down to a single "realloc". So far I have been able to successfully compile & run my above sample program without modification (except that I am no longer using libcxx, but my own C++ headers).

I think for now I may go down the route of just replacing all C++ specifics with my own super-simple implementation so that LLI doesn't choke itself. But I fear that as soon as I start incorporating the huge game library, there will be stuff that breaks even with all my own headers.

[Alon Zakai]

lli is not reliable, I don't think the LLVM devs put much effort into it. Instead, you can use the nativizer tool to create parallel js/native builds from the same bitcode, but it is limited - will work on small source files, has not been tested on huge complete programs, tools/nativize_llvm.py.

I assume you've already taken a look at https://github.com/kripken/emscripten/wiki/Debugging , since you say you tried SAFE_HEAP etc. Have you also build with -Werror? I added that to that wiki page now.

Worth verifying that SAFE_HEAP, assertions etc. were enabled (a typo in the command might lead to them not being activiated). Just for sanity's sake, seeing lots of SAFE_HEAP_LOAD commands in the output, and it running slowly, verifies it is on.

It's not likely that you've hit a mistranslated instruction, because others have not hit it, and we've run fuzzers, etc. Also, if there was a compiler error of that nature, it should only show up in some compiler modes - have you tried -O0, -O1, -O2? If the problem shows up in all, it's not likely to be a codegen issue. Instead, what is more likely is you are using part of libc or GL or another API that is not tested or properly supported. You can build with LIBRARY_DEBUG=1 to emit traces on every call to JS library code, which might help.

A possible but unlikely thing is a problem in your emscripten setup. You can rule that out by running the test suite, for example    python tests/runner.py asm2    will run a representative part; if that passes, you should be ok there.

If nothing else helps, the best route is to just find the first weird thing in the output - a misrendered image - and look at the commands that try to emit it, and debug backwards, as Jukka said. Adding printfs in the source and building natively and to the web, then comparing that, can let you focus until you find a significant difference.

- Alon

--

[Christoph Husse]

Okay thanks for that input. Gonna try it today. Maybe SAFE_HEAP was off because I didn't get any output.

Besides what you have recommended, I was working on "un-stding" my sources yesterday with success. There are NO standard library headers and also no memory allocations (everything froma  static buffer) and no conditional compilation macros... All types are the same size and sign on all platforms now. 

Only left is the big/little endian thing. Do you know if it is okay to do runtime code switches based on endianess? The code needs to take different paths depending on endianess... Its a very low level CPU emulator, so there is a lot of very platform specific low-level code in it.

 

[Jukka Jylänki]

Emscripten/JS is a little endian platform, just like X86, so the same code paths should work for JS and X86. It is ok to test for this at runtime and do different things based on the result. Just make sure that the JS platform correctly takes the little endian path and that should be all good there.

--

[Christoph Husse]

EWWW!!!

Looks like I have alignment issues (and well... my code too xD).

So I have now worked a lot on the sources to have the same types on
all platforms, compile by using a single "cpp" file, etc... And I also
got the emscripten options like SAFE_HEAP really to work.
After executing the thing with Node.js I see that I have unaligned
reads & writes. using UNALIGNED_MEMORY=1 fixes it but its slow because
it generates the machinery for all loads/stores while only few of them
are unaligned.

Is there a way to use UNALIGNED_MEMORY while at the same time listing
all the places where it actually encountered an unaligned load/store?
That would be really helpful because right now I would have to find an
offending line, recompile, restart, find the next offending line,
etc... This would be extremely time consuimg especially since one
iteration may well take over 5 minutes...

Any ideas?

> You received this message because you are subscribed to a topic in the
> Google Groups "emscripten-discuss" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/emscripten-discuss/tOz2Yc_sLuA/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[Jukka Jylänki]

Hmm, looking at the current code, as a quick manual fix you could try compiling with -s SAFE_HEAP=1, and then going to the output file and editing the generated functions SAFE_HEAP_LOAD and SAFE_HEAP_STORE to just 'console.error()' instead of 'abort()'. Does that work?

[Christoph Husse]

Well its a nice thought but the issue will most likely be that then
the application gets broken and doesn't do what it would normally do
;). I need the UNALIGNED_MEMORY fix in combination with a notification
for actual unaligned accesses.

[Christoph Husse]

As this was the initial problem. The application does not work without
UNALIGNED_MEMORY.

[Jukka Jylänki]

Oh, I misread Module.getValue and Module.setValue that they'd do unaligned accesses. Well, what happens if you combine that modification with -s UNALIGNED_MEMORY=1? Are those two incompatible?

[Christoph Husse]

Hmm I don't know :D. I will try it if my current approach fails. I
notices from what Emscripten reported so far that the CPU emulator I
am trying to compile conveniently pipes all memory accesses through
macros that do the little/big endian handling. So basically I am now
trying to hook a warn&fix template into those macros and hope that
there are not any other misaligned accesses done except through those
macros.

[Jukka Jylänki]

Asking around in #llvm irc, people say there that in Clang a syntax something like "uint32_t __attribute__((align(1))) *pointer = xxx;" could be used to signal that deferences through that particular pointer should be treated as unaligned. That should result in unaligned reference in the LLVM bytecode, which Emscripten is able to pick up and generate an unaligned load/store, even when no special link flags are used. That could be worth a try, e.g. typedeffing/defining a pointer type unaligned_u32_ptr or similar?

[Christoph Husse]

Hmm that also sounds interesting. I am going to try that too ;).

Besides that I have written my read_aligned & write_aligned templates
and now it works without UNALIGNED_MEMORY, doing the very same
"fallback" only at the places where memory actually could be
unaligned. Performance though is still not fully satisfactory, since
its just A LOT of places where those templates need to be used. So
maybe your Clang pointer attribute can get things done faster.

[Alon Zakai]

You can just hack in some code in SAFE_HEAP_LOAD/STORE to actually do unaligned loads and stores. We could add an option for that, in fact.

- Alon

[Christoph Husse]

You can just hack in some code in SAFE_HEAP_LOAD/STORE to actually do unaligned loads and stores. We could add an option for that, in fact.

Yeah that would be nice :).

 

In my case it doesn't really help though, since I can't avoid unaligned accesses. But there are only few places where unaligned access is possible so that's a special case for me. Luckily my described macro-based fix was enough to cover them all. I am not sure if the following is the fastest way to do it but that is what I am using to "align" unaligned accesses:

template<class T>

inline T __read_aligned_internal(void* ptr)

{

auto bPtr = (byte*)ptr;

T res = 0; 

switch (sizeof(T))

{

case 2:

res = (*bPtr) | (((T)*(bPtr+1)) << 8);

break;

case 4:

res = (*bPtr) | (((T)*(bPtr+1)) << 8) | (((T)*(bPtr+2)) << 16) | (((T)*(bPtr+3)) << 24);

break;

default:

throw "Unknown alignment.";

}

return res;

}

template<class T>

inline T __write_aligned_internal(void* ptr, T val)

{

auto bPtr = (byte*)ptr;

switch (sizeof(T))

{

case 2:

(*bPtr) = val & 0xFF;

(*(bPtr + 1)) = (val & 0xFF00) >> 8;

break;

case 4:

(*bPtr) = val & 0xFF;

(*(bPtr + 1)) = (val & 0xFF00) >> 8;

(*(bPtr + 2)) = (val & 0xFF0000) >> 16;

(*(bPtr + 3)) = (val & 0xFF000000) >> 24;

break;

default:

throw "Unknown alignment.";

}

return val;

}

 

[Christoph Husse]

But back to the general public... I think its an awesome idea to add this option you described. Because an application with misaligned accesses usually will not only contain one of them and it gets very tedious to figure them all out if SAFE_HEAP terminates your app on every occasion. Even further it might be possible to only report for each single line of SAFE_HEAP_LOAD etc ONCE per run, so that you don't get spammed with useless double reports. It's then easy to map the reported lines back to C++ sources with a debug info options as each SAFE_HEAP_LOAD will have the C++ code line as a comment behind it (could be done in a simple script for instance)...

As far as I know there is no tool outside of emscripten which allows you to enumerate unaligned accesses. Valgrind had a feature request but it seems it landed on the GTFO TODO list for whatever reason... 

[Chad Austin]

On Thu, May 8, 2014 at 11:06 PM, Christoph Husse <thesai...@googlemail.com> wrote:

As far as I know there is no tool outside of emscripten which allows you to enumerate unaligned accesses. Valgrind had a feature request but it seems it landed on the GTFO TODO list for whatever reason... 

You could just run your program on any other architecture than x86 and watch it explode on every unaligned access.  :)

[Christoph Husse]

You could just run your program on any other architecture than x86 and watch it explode on every unaligned access.  :)

xD that's what emscripten does too :P. I was referring to "enumerate"... so like enumerate all of them and then explode if need be. So you avoid the annoying test/fail/fix cycle which can be endless and especially with emscripten can take a lot of time (my project takes about 3 minutes to compile with emscripten atm) 

[Alon Zakai]

That hasn't been my experience, actually - when I've ported apps, they tended to have just a small amount of unaligned accesses (e.g. in network-reading code, serializing code, or GC code). Just rebuilding after fixing each one was fast enough. I'm surprised you have so many - what is their cause? Does your app purposefully pack structs to unaligned offsets or something like that? Generally speaking it isn't "easy" to cause an unaligned access in C/C++.

- Alon

On Thu, May 8, 2014 at 11:06 PM, Christoph Husse <thesai...@googlemail.com> wrote:

But back to the general public... I think its an awesome idea to add this option you described. Because an application with misaligned accesses usually will not only contain one of them and it gets very tedious to figure them all out if SAFE_HEAP terminates your app on every occasion. Even further it might be possible to only report for each single line of SAFE_HEAP_LOAD etc ONCE per run, so that you don't get spammed with useless double reports. It's then easy to map the reported lines back to C++ sources with a debug info options as each SAFE_HEAP_LOAD will have the C++ code line as a comment behind it (could be done in a simple script for instance)...

As far as I know there is no tool outside of emscripten which allows you to enumerate unaligned accesses. Valgrind had a feature request but it seems it landed on the GTFO TODO list for whatever reason... 

--

[Christoph Husse]

Well I am porting a CPU emulator and it seems to think unaligned
memory accesses are really cool, or well at least the programs that
run on this emulator :)

BTW, The port works now and runs almost good on Chrome... Like 100%
CPU utilization (one core). I think I can queeze out a bit more by
skipping some frames and using webworkers for some stuff too to get a
smooth experience.

But on Mozilla Firefox this port stinks. Like its a slideshow. Really bad.

> You received this message because you are subscribed to a topic in the
> Google Groups "emscripten-discuss" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/emscripten-discuss/tOz2Yc_sLuA/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to

[Alon Zakai]

Ah, CPU emulator makes some sense if it has a bytecode VM or something else with packed data.

That performance sounds very surprising. Is it on a fully optimized build? Do you see errors in the web console in firefox? It should warn if there is a perf problem like asm.js not validating.

- Alon

[Christoph Husse]

Meh damn you are right... There was something wrong with firefox ;).
Since the Web Audio API im using seems to be broken on Firefox, it
doesn't utilize much CPU because it seems to be stalled by some audio
stuff. Weird. I will have a look into it and keep you updated with the
performance when audio is active :).

[Christoph Husse]

Well damn xD. Firefox is twice as fast :O. Game plays totally smooth
there with only 50% CPU utilization...

Here can see for yourself if you like... Finally got it working I am
so happy lol:

https://dl.dropboxusercontent.com/u/27687180/WebServer/SNESOnline/SNESOnline.html

Input is:

Left, Right (steer), up, down (menu)
"A" for jump
"S" for use collectible
"D" for drive and advance menu

I think anyone who is older than 20 years knows this game anyway :D

Awesome job with emscripten. This is really impressive to myself.
Even though I have to say that the performance is still quite bad. On
my native C++ build it consumes about 30 times less CPU than in the
browser.


On Fri, May 9, 2014 at 10:13 PM, Christoph Husse

[Christoph Husse]

LOL firefox Nightly build is again about twice as fast :O.

On Fri, May 9, 2014 at 11:01 PM, Christoph Husse

[Jukka Jylänki]

Very nice!

I put this quickly through the Firefox profiler, which looks like this: https://dl.dropboxusercontent.com/u/40949268/emcc/snesonline.png . Based on that, the two biggest CPU consumers are in processing audio (inside the game code), and reading or writing the SDL canvas buffer between the game code and the 2D canvas. This SDL canvas upload code is known to be very slow (_SDL_LockSurface and _SDL_UnlockSurface), and not very optimized. A custom non-SDL-esque path of uploading the framebuffer to a 2D canvas might speed up here considerably.

To profile the audio performance, try adding -g2 flag, and disable javascript.options.asmjs preference in about:config in Firefox for the duration of the profile. That should allow the profiler to show frames inside the audio update function as well.

[Christoph Husse]

Oh thanks for your input :). Yeah this is just a first shot at it. I
have some funny plans with this thing and I will certainly come back
to performance improvements when I get the framework around it done.
Hope you are still there in one or two weeks :P. I have not much
experience with the browser environment at all, so I will need all the
help I can get to make this thing run smoothly :).

[Christoph Husse]

And btw, Audio is doing ALL the game processing. I emulate SNES when
generating audio samples as this automatically syncs it with audio ;).
So that's why this is so expensive. But the SDL part can probably be a
bit more optimized since right now it does two buffer copies (one from
SNES to SDL and then from SDL to canvas probably and maybe one more).

On Fri, May 9, 2014 at 11:22 PM, Christoph Husse