emcc working inconsistently

[Michel Rouzic]

I'm new to emscripten and I'm experiencing some extraordinary inconsistencies with emcc when trying to build hello_world.c. Here's an example, the commands are ran in quick succession with no outside changes:

C:\Program Files\Emscripten\emscripten\1.37.1>emcc tests/hello_world.c

C:\Program Files\Emscripten\emscripten\1.37.1>node a.out.js

C:\Program Files\Emscripten\emscripten\1.37.1>emcc tests/hello_world.c

C:\Program Files\Emscripten\emscripten\1.37.1>node a.out.js

C:\Program Files\Emscripten\emscripten\1.37.1>emcc tests/hello_world.c

C:\Program Files\Emscripten\emscripten\1.37.1>node a.out.js

hello, world!

so most of the time it doesn't seem to compile properly but sometimes it does? And without me changing anything, just trying again?

while other times I just get this:

C:\Program Files\Emscripten\emscripten\1.37.1>emcc tests/hello_world.c

Traceback (most recent call last):

  File "C:\Program Files\Emscripten\emscripten\1.37.1\\emcc", line 13, in <module>

    emcc.run()

  File "C:\Program Files\Emscripten\emscripten\1.37.1\emcc.py", line 1635, in run

    final = shared.Building.emscripten(final, append_ext=False, extra_args=extra_args)

  File "C:\Program Files\Emscripten\emscripten\1.37.1\tools\shared.py", line 1745, in emscripten

    call_emscripten(cmdline)

  File "C:\Program Files\Emscripten\emscripten\1.37.1\emscripten.py", line 1783, in _main

    temp_files.run_and_clean(lambda: main(

  File "C:\Program Files\Emscripten\emscripten\1.37.1\tools\tempfiles.py", line 78, in run_and_clean

    return func()

  File "C:\Program Files\Emscripten\emscripten\1.37.1\emscripten.py", line 1788, in <lambda>

    DEBUG=DEBUG,

  File "C:\Program Files\Emscripten\emscripten\1.37.1\emscripten.py", line 1689, in main

    temp_files=temp_files, DEBUG=DEBUG)

  File "C:\Program Files\Emscripten\emscripten\1.37.1\emscripten.py", line 93, in emscript

    glue, forwarded_data = compiler_glue(metadata, settings, libraries, compiler_engine, temp_files, DEBUG)

  File "C:\Program Files\Emscripten\emscripten\1.37.1\emscripten.py", line 293, in compiler_glue

    cwd=path_from_root('src'), error_limit=300)

  File "C:\Program Files\Emscripten\emscripten\1.37.1\tools\jsrun.py", line 122, in run_js

    raise Exception('Expected the command ' + str(command) + ' to finish with return code ' + str(assert_returncode) + ', but it returned with code ' + str(proc

.returncode) + ' instead! Output: ' + str(ret)[:error_limit])

Exception: Expected the command ['C:/Program Files/Emscripten/node/4.1.1_64bit/bin/node.exe', 'C:\\Program Files\\Emscripten\\emscripten\\1.37.1\\src\\compiler.

js', 'c:\\users\\user\\appdata\\local\\temp\\tmpqrxkb5.txt', 'C:\\Program Files\\Emscripten\\emscripten\\1.37.1\\src\\library_pthread_stub.js'] to finish with r

eturn code 0, but it returned with code 1 instead! Output: // The Module object: Our interface to the outside world. We import

// and export values on it, and do the work to get that through

// closure compiler if necessary. There are various ways Module can be used:

// 1. Not defined. We create it here

// 2. A function parameter, function(Module) { ..gener

WTF? I'm running it on Windows 7, it's a fresh install and this is what I get when I run emcc -v:

C:\Program Files\Emscripten\emscripten\1.37.1>emcc -v

emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 1.37.1

clang version 3.9.0  (emscripten 1.37.1 : 1.37.1)

Target: x86_64-pc-windows-msvc

Thread model: posix

InstalledDir: C:\Program Files\Emscripten\clang\e1.37.1_64bit

INFO:root:Checking JS engine ['C:/Program Files/Emscripten/node/4.1.1_64bit/bin/node.exe'] failed. Check your config file. Details: Expected the command ['C:/Pr

ogram Files/Emscripten/node/4.1.1_64bit/bin/node.exe', 'C:\\Program Files\\Emscripten\\emscripten\\1.37.1\\src\\hello_world.js'] to finish with return code 0, b

ut it returned with code 1 instead! Output:

INFO:root:(Emscripten: Running sanity checks)

CRITICAL:root:The JavaScript shell used for compiling (['C:/Program Files/Emscripten/node/4.1.1_64bit/bin/node.exe']) does not seem to work, check the paths in

C:\Users\user\.emscripten

The last line looks a bit worrying... I have no idea what's wrong, help. Thank you!

[Jukka Jylänki]

First off, can you try updating to Emscripten 1.37.13 or even the
incoming branch to see if that resolves the issue? ("emsdk install
latest" and "emsdk activate latest" or "emsdk install
sdk-incoming-64bit" and "emsdk activate sdk-incoming-64bit")

When the compiler produces working or failing builds, do they have
same or different contents and file sizes? Can you try diffing the
contents of good and bad builds to see if the differences are
something that could give any clues?

The contents of hello_world.js in Emscripten src directory should not
change. Try also running node hello_world.js directly on the command
line if that gives any errors. Also, experience says that generally
these types of inconsistencies are often caused by threading, so you
can try setting the environment variable "set EMCC_CORES=1" to force
Emscripten to run in singlethreaded mode to see if that would have any
effect on this type of nondeterminism.

> --
> You received this message because you are subscribed to the Google Groups
> "emscripten-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to emscripten-disc...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Floh]

This might not be very helpful, but I haven't seen this type of error on Windows yet. I have spent some time a few months ago to make my build system work on Windows for emscripten. 

I have seen similarly strange errors in the past (mostly on OSX) when running a parallel build after a new emscripten SDK install or SDK update, where the emscripten C runtime libs are compiled in the background, which happens after compiling and before asm.js code generation for executable targets. If several of those ran in parallel I saw all types of strange error messages and half-finished builds, but this is only for the first build after an SDK update.

This is different from your error though, since you are building sequentially and only a single executable.

The only other problem I ran into under Windows:

I was hitting the 8192 character command line length limit with long linker command lines, the only way around that was to use linker-response-files (https://gcc.gnu.org/wiki/Response_Files), which I only got working via cmake when targeting GNU Make, not Ninja as build tool, but it was not trivial to find a GNU Make tool which actually *works* on Windows in the normal command line environment (outside of Cygwin).

In the past I was simply running emscripten builds in a Linux VM, which was much easier to setup (and much faster!) then running the compilation on Windows (in Win10, the new Linux subsystem also might be an option). But once I got all Windows build issues resolved (which were actually build tool issues), it works quite well (but much slower than Linux or OSX, but that seems to be because clang is slower on Windows).

Cheers,

-Floh.

[Michel Rouzic]

On Sunday, June 11, 2017 at 10:44:52 AM UTC+2, jj wrote:

First off, can you try updating to Emscripten 1.37.13 or even the
incoming branch to see if that resolves the issue? ("emsdk install
latest" and "emsdk activate latest" or "emsdk install
sdk-incoming-64bit" and "emsdk activate sdk-incoming-64bit")

 

"emsdk install sdk-incoming-64bit" doesn't even work, it ends with this:

  gtest.vcxproj -> C:\Program Files\Emscripten\clang\fastcomp\build_incoming_vs2015_64\RelWithDebInfo\lib\gtest.lib

  gtest_main.vcxproj -> C:\Program Files\Emscripten\clang\fastcomp\build_incoming_vs2015_64\RelWithDebInfo\lib\gtest_main.lib

RC : fatal error RC1106: invalid option: -ologo [C:\Program Files\Emscripten\clang\fastcomp\build_incoming_vs2015_64\utils\PerfectShuffle\llvm-PerfectShuffle.vcxproj]

RC : fatal error RC1106: invalid option: -ologo [C:\Program Files\Emscripten\clang\fastcomp\build_incoming_vs2015_64\tools\llvm-config\llvm-config.vcxproj]

RC : fatal error RC1106: invalid option: -ologo [C:\Program Files\Emscripten\clang\fastcomp\build_incoming_vs2015_64\tools\llvm-cxxfilt\llvm-cxxfilt.vcxproj]

RC : fatal error RC1106: invalid option: -ologo [C:\Program Files\Emscripten\clang\fastcomp\build_incoming_vs2015_64\tools\llvm-mcmarkup\llvm-mcmarkup.vcxproj]

RC : fatal error RC1106: invalid option: -ologo [C:\Program Files\Emscripten\clang\fastcomp\build_incoming_vs2015_64\utils\not\not.vcxproj]

RC : fatal error RC1106: invalid option: -ologo [C:\Program Files\Emscripten\clang\fastcomp\build_incoming_vs2015_64\utils\yaml-bench\yaml-bench.vcxproj]

Build failed due to exception!

Working directory: C:/Program Files/Emscripten/clang/fastcomp\build_incoming_vs2015_64

Command '['C:\\Program Files (x86)\\MSBuild/14.0/Bin/amd64\\MSBuild.exe', '/t:Build', '/p:Configuration=RelWithDebInfo', '/p:Platform=x64', '/nologo', '/verbosity:minimal', 'LLVM.sln']' returned non-z

ero exit status 1

Installation failed!

"emsdk install latest" does work, and updates to 1.37.9 (yesterday I did the same and it updated to 1.37.1), but now when I try to compile anything, emcc gives me "CRITICAL:root:The JavaScript shell used for compiling (['C:/Program Files/Emscripten/node/4.1.1_64bit/bin/node.exe']) does not seem to work, check the paths in C:\Users\user\.emscripten" (epilogue: it does semi-work now as you will see below, although "emcc -v" still gives me that message). Am I supposed to install something? I only installed CMake.

When the compiler produces working or failing builds, do they have
same or different contents and file sizes? Can you try diffing the
contents of good and bad builds to see if the differences are
something that could give any clues?

It seems like when it fails it actually doesn't make a build at all.

 

The contents of hello_world.js in Emscripten src directory should not
change. Try also running node hello_world.js directly on the command
line if that gives any errors. Also, experience says that generally
these types of inconsistencies are often caused by threading, so you
can try setting the environment variable "set EMCC_CORES=1" to force
Emscripten to run in singlethreaded mode to see if that would have any
effect on this type of nondeterminism.

I tried setting the cores to 1 before updating however it was still acting as inconsistent.

Now have a look at this absolute mess, I ran these commands in quick succession:

C:\Program Files\Emscripten\emscripten\1.37.9>emcc tests/hello_world.c -o tests/hello_world.html

INFO:root:Checking JS engine ['C:/Program Files/Emscripten/node/4.1.1_64bit/bin/node.exe'] failed. Check your config file. Details: Expected the command ['C:/Program Files/Emscripten/node/4.1.1_64bit/

bin/node.exe', 'C:\\Program Files\\Emscripten\\emscripten\\1.37.9\\src\\hello_world.js'] to finish with return code 0, but it returned with code 1 instead! Output:

INFO:root:(Emscripten: Running sanity checks)

CRITICAL:root:The JavaScript shell used for compiling (['C:/Program Files/Emscripten/node/4.1.1_64bit/bin/node.exe']) does not seem to work, check the paths in C:\Users\user\.emscripten

C:\Program Files\Emscripten\emscripten\1.37.9>node faffa.js

C:\Program Files\Emscripten\emscripten\1.37.9>emcc tests/hello_world.c -o tests/hello_world.html

INFO:root:(Emscripten: Running sanity checks)

INFO:root:generating system library: libc.bc... (this will be cached in "C:\Users\user\.emscripten_cache\asmjs\libc.bc" for subsequent builds)

WARNING:root:Accessing the Emscripten cache at "C:\Users\user\.emscripten_cache" is taking a long time, another process should be writing to it. If there are none and you suspect this process has dead

locked, try deleting the lock file "C:\Users\user\.emscripten_cache.lock" and try again. If this occurs deterministically, consider filing a bug.

WARNING:root:Accessing the Emscripten cache at "C:\Users\user\.emscripten_cache" is taking a long time, another process should be writing to it. If there are none and you suspect this process has dead

locked, try deleting the lock file "C:\Users\user\.emscripten_cache.lock" and try again. If this occurs deterministically, consider filing a bug.

WARNING:root:Accessing the Emscripten cache at "C:\Users\user\.emscripten_cache" is taking a long time, another process should be writing to it. If there are none and you suspect this process has dead

locked, try deleting the lock file "C:\Users\user\.emscripten_cache.lock" and try again. If this occurs deterministically, consider filing a bug.

INFO:root: - ok

INFO:root:generating system library: dlmalloc.bc... (this will be cached in "C:\Users\user\.emscripten_cache\asmjs\dlmalloc.bc" for subsequent builds)

INFO:root: - ok

Traceback (most recent call last):

  File "C:\Program Files\Emscripten\emscripten\1.37.9\\emcc", line 13, in <module>

    emcc.run()

  File "C:\Program Files\Emscripten\emscripten\1.37.9\emcc.py", line 1673, in run

    final = shared.Building.emscripten(final, append_ext=False, extra_args=extra_args)

  File "C:\Program Files\Emscripten\emscripten\1.37.9\tools\shared.py", line 1963, in emscripten

    call_emscripten(cmdline)

  File "C:\Program Files\Emscripten\emscripten\1.37.9\emscripten.py", line 1852, in _main

    temp_files.run_and_clean(lambda: main(

  File "C:\Program Files\Emscripten\emscripten\1.37.9\tools\tempfiles.py", line 78, in run_and_clean

    return func()

  File "C:\Program Files\Emscripten\emscripten\1.37.9\emscripten.py", line 1857, in <lambda>

    DEBUG=DEBUG,

  File "C:\Program Files\Emscripten\emscripten\1.37.9\emscripten.py", line 1758, in main

    temp_files=temp_files, DEBUG=DEBUG)

  File "C:\Program Files\Emscripten\emscripten\1.37.9\emscripten.py", line 93, in emscript

    glue, forwarded_data = compiler_glue(metadata, settings, libraries, compiler_engine, temp_files, DEBUG)

  File "C:\Program Files\Emscripten\emscripten\1.37.9\emscripten.py", line 296, in compiler_glue

    cwd=path_from_root('src'), error_limit=300)

  File "C:\Program Files\Emscripten\emscripten\1.37.9\tools\jsrun.py", line 122, in run_js

    raise Exception('Expected the command ' + str(command) + ' to finish with return code ' + str(assert_returncode) + ', but it returned with code ' + str(proc.returncode) + ' instead! Output: ' + st

r(ret)[:error_limit])

Exception: Expected the command ['C:/Program Files/Emscripten/node/4.1.1_64bit/bin/node.exe', 'C:\\Program Files\\Emscripten\\emscripten\\1.37.9\\src\\compiler.js', 'c:\\users\\user\\appdata\\local\\t

emp\\tmptssywj.txt', 'C:\\Program Files\\Emscripten\\emscripten\\1.37.9\\src\\library_pthread_stub.js'] to finish with return code 0, but it returned with code 1 instead! Output: // The Module object:

 Our interface to the outside world. We import

// and export values on it, and do the work to get that through

// closure compiler if necessary. There are various ways Module can be used:

// 1. Not defined. We create it here

// 2. A function parameter, function(Module) { ..gener

C:\Program Files\Emscripten\emscripten\1.37.9>emcc tests/hello_world.c -o tests/hello_world.html

C:\Program Files\Emscripten\emscripten\1.37.9>node tests/hello_world.js

hello, world!

How does running "node" on a non-existent file (I wanted to see if it would say anything with a missing file) change the way emcc fails? So first it says node.exe doesn't work, then I try it again and it says "Exception: Expected the command" and then on the third try it compiles it without complaining about anything? Madness! Now I run the command over and over again and it totally randomly either fails like the second time (producing no file output) or succeeds with the hello_world.js working normally.

[Michel Rouzic]

I think you're right, I'll save myself a lot of trouble doing it on Linux. I set it up on an Ubuntu VM and it all compiled smoothly, so I think I'll stick to that! Thanks!