Emscripten 1.25.0 link problem (didn't happen in 1.22.0)

[Robert Goulet]

Hi all,

I just upgraded my project to Emscripten 1.25.0 and I have a new link error that wasn't there with Emscripten 1.22.0.

Here is what the linker outputs:

Link:
  Traceback (most recent call last):
    File "C:\Program Files\Emscripten\emscripten\1.25.0\emscripten.py", line 1578, in <module>
      _main(environ=os.environ)
    File "C:\Program Files\Emscripten\emscripten\1.25.0\emscripten.py", line 1566, in _main
      temp_files.run_and_clean(lambda: main(
    File "C:\Program Files\Emscripten\emscripten\1.25.0\tools\tempfiles.py", line 39, in run_and_clean
      return func()
    File "C:\Program Files\Emscripten\emscripten\1.25.0\emscripten.py", line 1574, in <lambda>
      DEBUG_CACHE=DEBUG_CACHE,
    File "C:\Program Files\Emscripten\emscripten\1.25.0\emscripten.py", line 1461, in main
      jcache=jcache, temp_files=temp_files, DEBUG=DEBUG, DEBUG_CACHE=DEBUG_CACHE)
    File "C:\Program Files\Emscripten\emscripten\1.25.0\emscripten.py", line 911, in emscript_fast
      %s''' % (mem_init.count(',')+1, global_initializers, mem_init)) # XXX wrong size calculation!
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 234337: ordinal not in range(128)
  Traceback (most recent call last):
    File "C:\Program Files\Emscripten\emscripten\1.25.0\\emcc", line 1259, in <module>
      final = shared.Building.emscripten(final, append_ext=False, extra_args=extra_args)
    File "C:\Program Files\Emscripten\emscripten\1.25.0\tools\shared.py", line 1464, in emscripten
      assert os.path.exists(filename + '.o.js'), 'Emscripten failed to generate .js'
  AssertionError: Emscripten failed to generate .js
C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Platforms\Emscripten\Microsoft.Cpp.Emscripten.Targets(117,5): error MSB6006: "C:\Program Files\Emscripten\emscripten\1.25.0\emcc.bat" exited with code 1.

Here is the link command-line use:

emcc -o f:/dev/build/main.html -O2 f:/dev/build/game/game.bc -Wno-warn-absolute-paths -s NO_EXIT_RUNTIME=1 -s TOTAL_MEMORY=64*1024*1024 --preload-file f:/dev/data/gamedata@/

I did a clean/full rebuild to be sure there wasn't any previously built files around and I always get the same results. Did anyone ever had this problem before? How do I debug this?

Thanks!

[Alon Zakai]

Hmm, never seen that before. Odd.

If this code is open, can you provide the bitcode for testing? If not, one random guess - if you change that line (911 in emscripten.py) to replace

mem_init

(in both places) with

str(mem_init)

, does that help?

- Alon

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Robert Goulet]

Nope, that didn't work...

Link:

  Traceback (most recent call last):

    File "C:\Program Files\Emscripten\emscripten\1.25.0\emscripten.py", line 1578, in <module>

      _main(environ=os.environ)

    File "C:\Program Files\Emscripten\emscripten\1.25.0\emscripten.py", line 1566, in _main

      temp_files.run_and_clean(lambda: main(

    File "C:\Program Files\Emscripten\emscripten\1.25.0\tools\tempfiles.py", line 39, in run_and_clean

      return func()

    File "C:\Program Files\Emscripten\emscripten\1.25.0\emscripten.py", line 1574, in <lambda>

      DEBUG_CACHE=DEBUG_CACHE,

    File "C:\Program Files\Emscripten\emscripten\1.25.0\emscripten.py", line 1461, in main

      jcache=jcache, temp_files=temp_files, DEBUG=DEBUG, DEBUG_CACHE=DEBUG_CACHE)

    File "C:\Program Files\Emscripten\emscripten\1.25.0\emscripten.py", line 911, in emscript_fast

      %s''' % (str(mem_init).count(',')+1, global_initializers, str(mem_init))) # XXX wrong size calculation!

  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 234337: ordinal not in range(128)

  Traceback (most recent call last):

    File "C:\Program Files\Emscripten\emscripten\1.25.0\\emcc", line 1259, in <module>

      final = shared.Building.emscripten(final, append_ext=False, extra_args=extra_args)

    File "C:\Program Files\Emscripten\emscripten\1.25.0\tools\shared.py", line 1464, in emscripten

      assert os.path.exists(filename + '.o.js'), 'Emscripten failed to generate .js'

  AssertionError: Emscripten failed to generate .js

C:\Program Files (x86)\MSBuild\Microsoft.Cpp\v4.0\Platforms\Emscripten\Microsoft.Cpp.Emscripten.Targets(117,5): error MSB6006: "C:\Program Files\Emscripten\emscripten\1.25.0\emcc.bat" exited with code 1.

Build FAILED.

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsub...@googlegroups.com.

[Robert Goulet]

Yeah unfortunately the code is private, can't share it. I wasn't able to reproduce it with a simple new project. The project that reproduce it links with about 20 static libraries (.bc), and it seems that if I remove about 3/4 of them, the link works. It doesn't seems to be tied to a specific lib, but more about the total size of the lib payload. That's very odd indeed.

[Bruce Mitchener]

Someone else reported this same bug a week or two ago on this list.

My suggestion to them is the same that I will make to you:

Modify the code to dump some of that value around the area that it is reporting the problem.  What's in position 234337 of mem_init? What's around there?

That should tell you something useful (and it is how I tracked down my problem when I ran into this in the past).

 - Bruce

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

[Robert Goulet]

Sure I would like to debug it. I have never debugged Emscripten before so I will need advices about how to do what you just said. Can you provide tips?

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-discuss+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Bruce Mitchener]

Sure.

Around that block of code, wrap it in a try / except block:

try:

  ...

except:

  print mem_init[234335:234339]

And I think that'll work. If that print errors, you may need to try printing mem_init[...].encode('utf-8') instead ...

 - Bruce

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

[Robert Goulet]

Ok I added the try/except block and I think it went into the except part but I can't see it; the print function does not write into the Visual Studio output window. So I tried to run the link command (emcc.bat with all the same parameters) and it still doesn't print the error in the console. How do I get this print function to write in the console (or VS output window) ?

[Bruce Mitchener]

I am half asleep but maybe:

print >> sys.stderr, mem_init...

http://stackoverflow.com/questions/5574702/how-to-print-to-stderr-in-python

 - Bruce

Sent from my iPhone

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

[Robert Goulet]

Cool that worked.

It prints 16,1 for 234335:234339

It seems the offending character is the comma?

There I printed 234300:234400 and I get:

103,104,116,95,100,105,114,101,99,116,105,111,110,0,58,0,0,0,0,0,0,0,115,108,105,99,101,115,0,0,100,

I really don't see what is wrong with these characters... encoding with utf-8 doesn't change the print output.

[Alon Zakai]

Converting those last bytes to a string gives "ght_direction\x00:\x00\x00\x00\x00\x00\x00\x00slices\x00\x00d". There isn't anything in emscripten, LLVM or clang with the characters "ght_direction". Perhaps look for where that is in your codebase? Might be you have a function with non-unicode characters in its name or in a parameter name, and somehow clang and LLVM preserve that oddness when compiling.

Another odd possibility is somehow emscripten.py (the file that crashes) got corrupted somehow, and contains a non-ascii character in the text processing right before that crash? Using a new copy of emscripten could rule that out.

- Alon

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

[Robert Goulet]

That's very cleaver, I didn't think to convert those to ascii.

Btw I completely uninstalled Emscripten 1.25.0 and re-installed it and the issue was still present. I uninstalled it again and re-installed 1.22.0 and everything works fine. So we can rule that one out. Now back to 1.25.0...

Definitively a string in our code, so I am guessing this must be a change in Clang since I wouldn't know how Emscripten could possibly introduce issues like that. I will have to dig into Clang's compiler changes/options to see if I can do anything about it. Its just annoying that the whole engine was running perfectly in 1.22.0 and now it won't even finish linking.

That file being processed by Emscripten in this case, is it only the .mem content (static variables initialization?) or does it also process the whole byte code?

[Marc]

It is maybe "right_direction" instead of "ght_direction".

Le Wed, 29 Oct 2014 12:55:22 -0700 (PDT),
Robert Goulet <robert.g...@gmail.com> a écrit :

> >>>>>>>>>> emscripten-disc...@googlegroups.com. For more

> >>>>>>>>>> options, visit https://groups.google.com/d/optout.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>> You received this message because you are subscribed to the
> >>>>>>> Google Groups "emscripten-discuss" group.
> >>>>>>> To unsubscribe from this group and stop receiving emails from
> >>>>>>> it, send an email to

> >>>>>>> emscripten-disc...@googlegroups.com. For more

> >>>>>>> options, visit https://groups.google.com/d/optout.
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>> You received this message because you are subscribed to the
> >>>>> Google Groups "emscripten-discuss" group.
> >>>>> To unsubscribe from this group and stop receiving emails from
> >>>>> it, send an email to

> >>>>> emscripten-disc...@googlegroups.com. For more

> >>>>> options, visit https://groups.google.com/d/optout.
> >>>>>
> >>>>
> >>>> --
> >>> You received this message because you are subscribed to the
> >>> Google Groups "emscripten-discuss" group.
> >>> To unsubscribe from this group and stop receiving emails from it,

> >>> send an email to emscripten-disc...@googlegroups.com.

> >>> For more options, visit https://groups.google.com/d/optout.
> >>>
> >>> --
> >> You received this message because you are subscribed to the Google
> >> Groups "emscripten-discuss" group.
> >> To unsubscribe from this group and stop receiving emails from it,

> >> send an email to emscripten-disc...@googlegroups.com
> >> <javascript:>. For more options, visit
> >> https://groups.google.com/d/optout.
> >>
> >
> >
>

[Alon Zakai]

Not sure what you mean exactly, but in the python script, it is handling output from the JS backend, which handled both static initialization and executable code, basically everything.

If it works on 1.22.0 but not on 1.25.0, one option is to bisect to find the breaking version (or better, commit).

Another option is to reduce it manually, link less and less things in the final link (might need -s LINKABLE=1 to avoid dce killing everything once you remove the main() method), then continue narrowing it down to a single method, which hopefully you can provide a public testcase based on that.

- Alon

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.

[Robert Goulet]

Unfortunately I won't be able to dig into this for the next two-three weeks, but I will try to find the offending commit once I'm back. Thanks!

Sent from mobile, please excuse typos.

You received this message because you are subscribed to a topic in the Google Groups "emscripten-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/emscripten-discuss/Kfn59Uv8MBs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to emscripten-disc...@googlegroups.com.

[Aidan Hobson Sayers]

I'm unconvinced about the the focus on mem_init - unless the backend output is very confused, mem_init is a literal string like

/* memory initializer */ allocate([70,83,[...SNIP...],0,0,0,0], "i8", ALLOC_NONE, Runtime.GLOBAL_BASE);

and so shouldn't have strange bytes in it. I don't think decoding the js array in the python string tells us much either because regardless of what it comes to, it's not going to have any impact on Python.

global_initializers is from json deserialisation so should be reasonably safe (and I doubt clang would choose unicode names for initialisers).

Which leaves us with pre, which can be described as "the bit before 'EMSCRIPTEN_END_FUNCS'", per pre, post = glue.split('// EMSCRIPTEN_END_FUNCS')

Which seems to imply the bad byte is in an emscripten library somewhere.

It just so happens I got rid of a '\xc2\xa0' byte sequence from an emscripten library recently - https://github.com/kripken/emscripten/pull/2904

This fix got into 1.26.0 - can you try that?

Thanks
Aidan

[Robert Goulet]

emsdk update then emsdk list only shows 1.25.0 as the latest, is there a different way to get 1.26.0?

You received this message because you are subscribed to a topic in the Google Groups "emscripten-discuss" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/emscripten-discuss/Kfn59Uv8MBs/unsubscribe.

To unsubscribe from this group and all its topics, send an email to emscripten-discuss+unsub...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Aidan Hobson Sayers]

You could go into the emscripten directory, git pull and then git checkout 1.26.0, but the simpler solution is probably just to download https://raw.githubusercontent.com/kripken/emscripten/9d1cc3667b4805b90cbab01cb24f9776f71bc011/src/library_glfw.js (i.e. the file immediately after the patch was merged) and replace src/library_glfw.js with it.

To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.