core dump analysis & debugging

[Dan Milon]

I was wondering if it is possible to use a core dump from a node
process running on linux as a post mortem debugging mechanism.

Would smartos' MDB understand a core dump coming from a linux machine?

Thanks,
danmilon.

[Evan]

Slightly off topic, but I've been debugging some memory leaks in a node app, and after trying out quite a few heap and GC inspector packages, I've fallen in love with https://github.com/c4milo/node-webkit-agent.  It's a one-liner require that lets you use chrome's built-in JS debugger for memory and CPU consumption... and it it works for node 0.8.x!

[Ben Noordhuis]

The people at Joyent added V8 smarts to mdb as a plugin but I don't
know if it still works.

It's possible that mdb understands Linux core dumps but I expect it
doesn't. A core dump is an ELF file in disguise and about as
portable.

On a tangential note, there's built-in gdbjit support. It's not
perfect but it's functional enough to print JS function names in stack
traces.

$ ./configure --debug --gdb
$ make
<snip>
$ gdb --args out/Debug/node --gdbjit tmp/net-connect.js
(gdb) break uv_tcp_connect
Breakpoint 1 at 0xb92ca0: file ../../deps/uv/src/uv-common.c, line 245.
(gdb) run
Starting program: /home/bnoordhuis/src/nodejs/master/out/Debug/node
--gdbjit tmp/net-connect.js
Breakpoint 1, uv_tcp_connect (req=0x13870c0, handle=0x137a690,
address=..., cb=0x72efa6 <node::TCPWrap::AfterConnect(uv_connect_s*,
int)>) at ../../deps/uv/src/uv-common.c:245
245 if (handle->type != UV_TCP || address.sin_family != AF_INET) {
(gdb) bt
#0 uv_tcp_connect (req=0x13870c0, handle=0x137a690, address=...,
cb=0x72efa6 <node::TCPWrap::AfterConnect(uv_connect_s*, int)>) at
../../deps/uv/src/uv-common.c:245
#1 0x000000000072f3d4 in node::TCPWrap::Connect (args=...) at
../../src/tcp_wrap.cc:379
#2 0x00000000007c1cea in v8::internal::HandleApiCallHelper<false>
(args=..., isolate=0x1300070) at ../../deps/v8/src/builtins.cc:1372
#3 0x00000000007bc634 in v8::internal::Builtin_Impl_HandleApiCall
(args=..., isolate=0x1300070) at ../../deps/v8/src/builtins.cc:1390
#4 0x00000000007bc605 in v8::internal::Builtin_HandleApiCall
(args=..., isolate=0x1300070) at ../../deps/v8/src/builtins.cc:1389
#5 0x000002a75ec0654e in ?? ()
#6 0x000002a75ec150b8 in ?? ()
#7 0x000002a75ec064a1 in ?? ()
#8 0x00007fffffffd3d0 in ?? ()
#9 0x00007fffffffd438 in ?? ()
#10 0x000002a75ec6f9b8 in connect (self=..., address=..., port=...,
addressType=..., localAddress=...) at net.js:703
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

[Dan Milon]

Thanks, I'll give it a try.

Now, about getting an actual core dump, docs [1] say process.abort
will generate one.

$ node -e 'process.abort()'
Aborted (core dumped)

$ ls -a
./ ../

Should I file this as a bug?

[1] http://nodejs.org/api/process.html#process_process_abort

danmilon.

[Ben Noordhuis]

On Fri, Feb 1, 2013 at 4:38 PM, Dan Milon <danm...@gmail.com> wrote:
> Thanks, I'll give it a try.
>
> Now, about getting an actual core dump, docs [1] say process.abort
> will generate one.
>
> $ node -e 'process.abort()'
> Aborted (core dumped)
>
> $ ls -a
> ./ ../
>
> Should I file this as a bug?
>
> [1] http://nodejs.org/api/process.html#process_process_abort
>
> danmilon.

You have to set `ulimit -c unlimited` first.

[Dan Milon]

$ ulimit -c unlimited

$ node -e 'process.abort()'
Aborted (core dumped)

$ ls -a
./ ../

Any idea?

BTW, could you point me to the actual code of process.abort in the source?

danmilon.

[Isaac Schlueter]

On Fri, Feb 1, 2013 at 3:21 AM, Ben Noordhuis <in...@bnoordhuis.nl> wrote:
> The people at Joyent added V8 smarts to mdb as a plugin but I don't
> know if it still works.

It does. I use it from time to time. You may need to get on the
latest SmartOS build to get a v8.so that works with node v0.8.

> It's possible that mdb understands Linux core dumps but I expect it
> doesn't. A core dump is an ELF file in disguise and about as
> portable.

I've tried mdb'ing linux core dumps, and it has never gone well.
Maybe there's an incantation to make it work, but if so, I don't know
what it is.

[Ben Noordhuis]

On Fri, Feb 1, 2013 at 5:27 PM, Dan Milon <danm...@gmail.com> wrote:
> $ ulimit -c unlimited
>
> $ node -e 'process.abort()'
> Aborted (core dumped)
>
> $ ls -a
> ./ ../
>
> Any idea?
>
> BTW, could you point me to the actual code of process.abort in the source?
>
> danmilon.

Check what the kernel.core_pattern sysctl is set to. Some security
frameworks change it so core dumps get stored elsewhere.

Also, don't run as root. Some of the aforementioned frameworks block
root core dumps as a security precaution (think symlink attacks.)

process.abort() is implemented in src/node.cc (grep for 'Abort'). It
doesn't do anything special: it just calls abort() which in turn sends
a SIGABRT to the process.

[Dan Milon]

Indeed, systemd does some sneaky tricks.

$ sysctl kernel.core_pattern

kernel.core_pattern = |/usr/lib/systemd/systemd-coredump %p %u %g %s %t %e

$ sudo sysctl kernel.core_pattern='core'

$ node -e 'process.abort()'
Aborted

$ ls -a
./ ../ core


Thanks!