Can't make sense of segfault stack trace

[Arve Knudsen]

Hi

I've built ChucK with -g4 (thus producing a sourcemap) and -s SAFE_HEAP=3, and unsafe memory access is detected while executing ChucK. However, I am unable to debug the segfault and could use your help. I tell Chrome to halt on exceptions, and when exception halts on the segfault, a stack trace is produced that looks as follows:

    invoke_iiiii (chuck.js:9199)

    _executeCode (__tree:274)

    asm._executeCode (chuck_emit.cpp:915)

    ccallFunc (chuck.js:534)

    (anonymous function) (whole.html:1)

    jQuery.event.dispatch (jquery-1.10.2.js:5095)

    jQuery.event.add.elemData.handle (jquery-1.10.2.js:4766)

However, this stacktrace is of little use since it really doesn't make much sense. When I click on the asm._executeCode frame, the debugger enters the C++ function emit_engine_emit_do_while (line 915 of chuck.emit.cpp). Said line looks like this:

    while( emit->code->stack_cont.size() && emit->code->stack_cont.back() )

First of all, there's no correspondence betwen this function and the name of the stack frame ('asm._executeCode'). Second, when I click on the next frame, it resolves to the function __tree_balance_after_insert (of file '__tree'). This doesn't make much sense does it??

When I let execution continue after the exception, a completely different looking traceback is printed to the console:

    Uncaught abort() at Error

        at jsStackTrace (http://localhost:8000/examples/basic/chuck.js:1049:13)

        at stackTrace (http://localhost:8000/examples/basic/chuck.js:1066:22)

        at abort (http://localhost:8000/examples/basic/chuck.js:561343:25)

        at SAFE_HEAP_LOAD (http://localhost:8000/examples/basic/chuck.js:429:87)

        at _strlen (http://localhost:8000/examples/basic/chuck.js:552513:43)

        at Array.__ZNSt3__111char_traitsIcE6lengthEPKc [std::__1::char_traits<char>::length(char const*)] (http://localhost:8000/examples/basic/chuck.js:541966:7)

        at Object.dynCall_ii (http://localhost:8000/examples/basic/chuck.js:553175:74)

        at invoke_ii (http://localhost:8000/examples/basic/chuck.js:8873:32)

        at Array.__ZN12_GLOBAL__N_19sporkCodeEP14Chuck_CompilerP8Chuck_VMPKcS5_ [(anonymous namespace)::sporkCode(Chuck_Compiler*, Chuck_VM*, char const*, char const*)] (http://localhost:8000/examples/basic/chuck.js:256286:8)

        at Object.dynCall_iiiii (http://localhost:8000/examples/basic/chuck.js:551097:77)

How can I debug this exception?? Please help.

If you like, you can reproduce by:

1. Check out https://github.com/aknuds1/chuck.git

2. cd chuck

3. pushd src && make -j8 CHUCK_DEBUG_LEVEL=4 CHUCK_EM_SAFEHEAP=3 emscripten && popd && pushd js && grunt

4. python -m SimpleHTTPServer

5. Open http://localhost:8000/examples/basic/whole.html in Chrome (evt. some other browser)

6. Open the Developer Tools

7. Enable 'Pause on exceptions'

8. Click the 'Play' button

Thanks,

Arve

[Arve Knudsen]

I forgot to mention that before running grunt, you must've installed it globally (npm install -g grunt-cli) and then run 'npm install' within the 'js' directory.

Arve

[Alon Zakai]

Any chance you can make it possible to run just the emscripten part of the build process? E.g. running 'make' immediately fails on not having 'bison' installed. Also I'd rather not install grunt and other things globally.

- Alon

--
You received this message because you are subscribed to the Google Groups "emscripten-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emscripten-disc...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Arve Knudsen]

I'll think of something.

[Arve Knudsen]

Alon, would you be OK with using Docker to test?

Arve

On Sat, Dec 13, 2014 at 1:52 AM, Alon Zakai <alon...@gmail.com> wrote:

[Arve Knudsen]

I have a Docker/Fig config for you, but alas it won't build due to an internal compiler exception when installing Emscripten itself. See issue https://github.com/kripken/emscripten/issues/3078.

Arve

[Arve Knudsen]

Nevermind about the internal compiler error, I was compiling on a machine with too little memory, sorry about that. Am working on the Docker/fig config now.

Arve

[Alon Zakai]

I actually don't know anything about docker. Is it trivial to set up (apt-get)?

- Alon

[Arve Knudsen]

Yeah, you can install it easily via apt-get. I followed the instructions to install the very latest version, as the Ubuntu included version was a bit old: https://docs.docker.com/installation/ubuntulinux/. Fig is another tool, which you should be able to install via the Python package manager pip (http://www.fig.sh/install.html). I think I have a working Fig setup now, but haven't figured out how to reach it from the browser yet.

I did run into a potential problem, which was that I wasn't able to build ChucK on Linux with -g4, only -g, since clang didn't accept the -g4 flag. Is -g4 necessary to produce source maps?

Arve

[Alon Zakai]

-g4 is needed for source maps, but this can probably be debugged without them.

- Alon

[Arve Knudsen]

Alon, if you check out https://github.com/aknuds1/chuck and run the following: 'docker build -t chuckdemos . && docker run -p 8000:8000 chuckdemos', you should be able to open a ChucK demo at http://localhost:8000/js/examples/basic/whole.html. After loading said demo, click 'Play' on the page. You should see in the JavaScript console that there's been a memory access error. If this doesn't work for you, please let me know.

Thanks,

Arve

[Arve Knudsen]

Alon: Have you tried my Docker setup yet?

Thanks,

Arve

[Alon Zakai]

I tried now, but something is wrong on my system,

$ sudo apt-get install docker
Reading package lists... Done
Building dependency tree      
Reading state information... Done
The following NEW packages will be installed:
  docker
0 upgraded, 1 newly installed, 0 to remove and 6 not upgraded.
Need to get 12.2 kB of archives.
After this operation, 65.5 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/universe docker amd64 1.5-1 [12.2 kB]
Fetched 12.2 kB in 0s (39.6 kB/s)  
Selecting previously unselected package docker.
(Reading database ... 420485 files and directories currently installed.)
Preparing to unpack .../docker_1.5-1_amd64.deb ...
Unpacking docker (1.5-1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Setting up docker (1.5-1) ...

yet

$ docker
The program 'docker' is currently not installed. You can install it by typing:
sudo apt-get install docker

Is docker used in some special way that I am not doing?

- Alon

[Arve Knudsen]

Try following the instructions at https://docs.docker.com/installation/ubuntulinux/, you might have installed an older conflicting package named docker.

[Alon Zakai]

Thanks, that fixes the install issue. After some fiddling with users and groups, I now get

$ docker build -t chuckdemos . && docker run -p 8000:8000 chuckdemos
Sending build context to Docker daemon 99.34 MB
Sending build context to Docker daemon
Step 0 : FROM aknudsen/emscripten:1.28.0
 ---> 3bb870c168d4
Step 1 : RUN rm /bin/sh && ln -s /bin/bash /bin/sh
 ---> Using cache
 ---> a9a395a9e6f0
Step 2 : ADD . /code
 ---> Using cache
 ---> 16b7606b2d29
Step 3 : WORKDIR /code
 ---> Using cache
 ---> 88e18692a27f
Step 4 : RUN pushd /emsdk_portable && source ./emsdk_env.sh && popd && pushd src &&make clean && make -j5 CHUCK_DEBUG=1 CHUCK_EM_SOURCEMAP=1CHUCK_EM_SAFEHEAP=3 emscripten && popd && pushd js && npm install && grunt
 ---> Running in 186fecc685d1
/emsdk_portable /code
Adding directories to PATH:
PATH += /emsdk_portable

/code
/code/src /code
bison -dv -b chuck chuck.y
flex -ochuck.yy.c chuck.lex
make: emcc: Command not found
make: emcc: Command not found
make: *** [util_math.o] Error 127
make: *** Waiting for unfinished jobs....
make: *** [util_network.o] Error 127
emcc -I. -Ilo -g  -c util_math.c -o util_math.o
emcc -I. -Ilo -g  -c util_network.c -o util_network.o
2015/01/05 16:42:42 The command [/bin/sh -c pushd /emsdk_portable && source ./emsdk_env.sh && popd && pushd src &&make clean && make -j5 CHUCK_DEBUG=1 CHUCK_EM_SOURCEMAP=1CHUCK_EM_SAFEHEAP=3 emscripten && popd && pushd js && npm install && grunt] returned a non-zero code: 2
$
$
$ emcc
WARNING  root: no input files

It can't find emcc. I assume it looks inside the docker container and not in my system, but just in case I misunderstood something, I verified that emcc is on the path on my system.

- Alon

[Arve Knudsen]

Thanks for trying, Alon. Ugh, I thought the Docker image would just work. The Docker image is supposed to have emcc added to $PATH, don't know why it wouldn't work for you. Will have to investigate.

Arve

Best,

Arve

[Alon Zakai]

Another option is to make a repo with the final bitcode file + other necessary files for the bitcode => js stage, + a one-line command to do the build. That should be enough for me to see what's going on (although without the C source, somewhat harder).

- Alon

[Arve Knudsen]

The Docker image was working flawlessly for me so there must be some tiny niggle.

[Arve Knudsen]

I see a difference between your Docker output and mine. Yours says

  Adding directories to PATH:
  PATH += /emsdk_portable

Whereas mine says:

  Adding directories to PATH:

  PATH += /emsdk_portable

  PATH += /emsdk_portable/clang/fastcomp/build_master_64/bin

  PATH += /emsdk_portable/emscripten/master

Any idea why emsdk_env.sh would not add the last two directories to PATH in your case? Which OS are you using? I'm using Ubuntu 64-bit 14.10 and Docker 1.4.1.

Arve

[Alon Zakai]

No idea.

I'm on Ubuntu 14.04, 64-bit, Docker 1.0.1.

- Alon

[Arve Knudsen]

I may have fixed it, please pull my latest changes to the repository and build again.

[Arve Knudsen]

Your Docker looks really old though, I have version 1.4.1 Could you try installing the newest version, as described in section "Docker-maintained Package Installation" at https://docs.docker.com/installation/ubuntulinux/#ubuntu-trusty-1404-lts-64-bit?

Thanks,

Arve

[Alon Zakai]

I'd rather not install something not in the default repos, it always ends up causing problems later in my experience...

Trying on the latest, I get

$ docker build -t chuckdemos .

Sending build context to Docker daemon 99.35 MB

Sending build context to Docker daemon

Step 0 : FROM aknudsen/emscripten:1.29.0
 ---> 5e486031a224

Step 1 : RUN rm /bin/sh && ln -s /bin/bash /bin/sh

 ---> Running in c31daac066a0
2015/01/09 14:54:23 /var/lib/docker/aufs/mnt/c31daac066a0ad81f876a6ed2f943dac79d5f0bbc420d72c3fe41b9a17b57b02 is not within /var/lib/docker/aufs/mnt/c31daac066a0ad81f876a6ed2f943dac79d5f0bbc420d72c3fe41b9a17b57b02

Although it doesn't say so, I assume that last line is an error, as

$ docker run -p 8000:8000 chuckdemos
Unable to find image 'chuckdemos' locally
Pulling repository chuckdemos
2015/01/09 14:54:26 HTTP code: 404

- Alon

[Arve Knudsen]

Couldn't you just fire up a VM and install the latest Docker in there? I've never seen that kind of problem in my Docker use. It should save time in the long run, and probably just work. That's what I do (on my Mac) anyway.

You're not running out of disk space or something though? You might want to prune old Docker containers and images (https://www.calazan.com/docker-cleanup-commands/).

Arve

[Arve Knudsen]

Might be smart to prune old Docker containers/images first in any case, might get rid of that error if you're lucky.

[Alon Zakai]

Isn't Docker like a VM? I don't understand what running it in another VM would buy here... seems like a bootstrapping problem ;) Perhaps it makes installing a different version easier, but we don't know the version is the issue?

I tried to prune etc. with that link, then ran the commands again, same error.

At this point, I think Docker isn't working out. Unless someone else reading this wants to jump in and help. Otherwise, my suggestion is instead

1. Create 3 builds with: -O2 -profiling  ;   -O2 -profiling -s ASSERTIONS=1    ;    -O2 -profiling -s ASSERTIONS=2 -s SAFE_HEAP=1  
2. If you don't see the problem with lower optimizations than -O2, then also builds with same but with the highest lower optimization level that works
3. Put those in a repo I can check out

4. Also put the source code in that repo

5. List of instructions to see the problem in those builds

- Alon

[Arve Knudsen]

I mean, if you use a VM you shouldn't need to fear installing a new version of Docker. I'm using Ubuntu 14.10 64bit and I've never seen anything close to the issues you're seeing. It's quite possible they've fixed these issues with newer versions, as I was reading about stale data historically.

Try removing all chuckdemos images though, since I know that the dangling filter will leave tagged images behind.

[Arve Knudsen]

I also made sure to install Docker 1.4 myself due to it having a feature I needed, but I can't recall which now. So please just try the latest version. If you run it in a VM you'll also get rid of any bad state as a bonus.

[Alon Zakai]

The problem is then I need to install a VM, and make sure it works properly, which is about as hard as setting up Docker - in my experience they don't just work.

I honestly think the fastest path here is what I suggested before. If you make those builds, I can probably take a close look at those and figure out what is wrong.

- Alon
.