Can't crack core file from go v1.13.7

58 views
Skip to first unread message

Shaun Crampton

unread,
May 29, 2020, 10:57:11 AM5/29/20
to delve-dev
Hi, I tried to use delve to crack a corefile generated from my app, built with go v1.13.7.  I get a partial stack trace of the active goroutine but trying to list goroutines panics:

$ dlv core bin/calico-felix /tmp/core_felix-0-475288-2-felixfv_calico-felix.8
Type 'help' for list of commands.
(dlv) bt
 0  0x0000000000460d41 in runtime.raise
    at /usr/local/go-cgo/src/runtime/sys_linux_amd64.s:150
 1  0x0000000000446b3b in runtime.dieFromSignal
    at /usr/local/go-cgo/src/runtime/signal_unix.go:428
 2  0x00000000004470bd in runtime.sigfwdgo
    at /usr/local/go-cgo/src/runtime/signal_unix.go:631
 3  0x0000000000446220 in runtime.sigtrampgo
    at /usr/local/go-cgo/src/runtime/signal_unix.go:289
 4  0x0000000000461033 in runtime.sigtramp
    at /usr/local/go-cgo/src/runtime/sys_linux_amd64.s:357
 5  0x00007f1c026ad730 in ???
    at ?:-1
 6  0x0000000000446cda in runtime.crash
    at /usr/local/go-cgo/src/runtime/signal_unix.go:520
 7  0x00000000004317e4 in runtime.fatalpanic
    at /usr/local/go-cgo/src/runtime/panic.go:874
 8  0x00000000004311a2 in runtime.gopanic
    at /usr/local/go-cgo/src/runtime/panic.go:722
 9  0x000000000052acd2 in github.com/sirupsen/logrus.Entry.fireHooks
10  0x000000c0000e4120 in ???
    at ?:-1
12  0x000000c0000da030 in ???
    at ?:-1
14  0x00000000018fe47e in ???
    at ?:-1
15  0x000000000045f371 in runtime.goexit
    at /usr/local/go-cgo/src/runtime/asm_amd64.s:1357
(dlv) grs
Command failed: Internal debugger error: runtime error: invalid memory address or nil pointer dereference
runtime.gopanic (0x42f271)
/usr/local/go/src/runtime/panic.go:679
runtime.panicmem (0x4443db)
/usr/local/go/src/runtime/panic.go:199
runtime.sigpanic (0x444218)
/usr/local/go/src/runtime/signal_unix.go:394
reflect.Value.call (0x4bef65)
/usr/local/go/src/reflect/value.go:460
reflect.Value.Call (0x4be723)
/usr/local/go/src/reflect/value.go:321
runtime.goexit (0x45cb40)
/usr/local/go/src/runtime/asm_amd64.s:1357

(dlv) exit


The corefile was generated by running with GOTRACEBACK=panic and explicitly calling panic() (just to test core file collection before I use it in anger) so the process should have been in a good state.  I tried a couple of times with the same result.

-Shaun

chainhelen

unread,
May 31, 2020, 10:56:46 PM5/31/20
to delve-dev
Can you provide the coredump/binary file or the Minimum recurrence file?

在 2020年5月29日星期五 UTC+8下午10:57:11，Shaun Crampton写道：

Shaun Crampton

unread,
Jun 1, 2020, 4:54:09 AM6/1/20
to delve-dev
Aha, I tried to package up the binary and the core file by putting them in the same directory.  Before using tar, I tried dlv one last time and found that it worked if the core file and binary were both in the current directory.  

This works:

cp bin/calico-felix-amd64 /tmp/calico-felix
cd /tmp
dlv core calico-felix ./core_felix-0-16339-2-felixfv_calico-felix.7



but this doesn't:

dlv core bin/calico-felix-amd64 /tmp/core_felix-0-12988-2-felixfv_calico-felix.7

alessandr...@gmail.com

unread,
Jun 2, 2020, 12:04:41 PM6/2/20
to delve-dev, Shaun Crampton
See if https://github.com/go-delve/delve/pull/2070 fixes your problem. Also
check the value of `ulimit -c`.
> --
> You received this message because you are subscribed to the Google Groups "delve-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to delve-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/delve-dev/f4bb410d-5d7a-4672-bfd7-33e61afedc1f%40googlegroups.com.
>

Shaun Crampton

unread,
Jun 2, 2020, 12:23:56 PM6/2/20
to delve-dev
ulimit -c was unlimited.

I don't see the problem if the core file and the binary are in the current directory; suggests an issue with finding the file?
> To unsubscribe from this group and stop receiving emails from it, send an email to delv...@googlegroups.com.

alessandr...@gmail.com

unread,
Jun 2, 2020, 12:43:38 PM6/2/20
to delve-dev, Shaun Crampton
That's not possible. There's no reason why the path to the file
would make a difference for this. I notice that in your command
line examples you used two different core files, presumably only
one of them exhibits this behavior.
> > an email to delv...@googlegroups.com <javascript:>.
> > > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/delve-dev/f4bb410d-5d7a-4672-bfd7-33e61afedc1f%40googlegroups.com.
> >
> > >
> >
>
> --
> You received this message because you are subscribed to the Google Groups "delve-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to delve-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/delve-dev/7f322a28-0f95-4ac9-824b-34030fd02d16%40googlegroups.com.
>

Shaun Crampton

unread,
Jun 3, 2020, 10:32:12 AM6/3/20
to delve-dev
Think I must have confused myself and retested with the wrong corefile or one not matching the binary.

I can't seem to reproduce the problem now and I've reconfigured our CI system to save off the exact binary under test as well as the core files.
> > > reflect.Value.call (0x4bef65)
> > > /usr/local/go/src/reflect/value.go:460
> > > reflect.Value.Call (0x4be723)
> > > /usr/local/go/src/reflect/value.go:321
> > >
> > github.com/go-delve/delve/service/rpccommon.(*ServerImpl).serveJSONCodec.func2
> > > (0xa193fa)
> > > /home/shaun/go-os/pkg/mod/
> > > runtime.goexit (0x45cb40)
> > > /usr/local/go/src/runtime/asm_amd64.s:1357
> > >
> > > (dlv) exit
> > >
> > >
> > > The corefile was generated by running with GOTRACEBACK=panic and
> > explicitly
> > > calling panic() (just to test core file collection before I use it in
> > > anger) so the process should have been in a good state.  I tried a
> > couple
> > > of times with the same result.
> > >
> > > -Shaun
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > Groups "delve-dev" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > an email to delv...@googlegroups.com <javascript:>.
> > > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/delve-dev/f4bb410d-5d7a-4672-bfd7-33e61afedc1f%40googlegroups.com.
> >
> > >
> >
>
> --
> You received this message because you are subscribed to the Google Groups "delve-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to delv...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages