CGO core dump analysis using GDB

[mariappan balraj]

Hi,

I have following programming where making CGO from GO code. test3() is called from Go code. Which calls test2() and which calls test1(). In test1(), there is a NULL pointer assignment. I could able to generate the core dump when running the program. But when I use gdb, I am not getting the stack trace. Can someone please help?

package main

/*

void test1(void) {

   int *p = (int*)NULL;

   *p = 30;

}

void test2(void) { 

    test1();

}

void test3(void) { 

    test2();

}

*/

import "C"

func main() {

    C.test3();

}

Best Regards

Mariappan

[mariappan balraj]

Hello Go Experts,

Can someone please help on this? When we invoke CGO and there is a crash in C code, is it possible to get the complete C stack details to debug the problem? I am not seeing much information regarding this.

Best Regards

Mariappan

[Ian Lance Taylor]

On Thu, Dec 22, 2022 at 1:57 PM mariappan balraj
<mariappa...@gmail.com> wrote:
>
> I have following programming where making CGO from GO code. test3() is called from Go code. Which calls test2() and which calls test1(). In test1(), there is a NULL pointer assignment. I could able to generate the core dump when running the program. But when I use gdb, I am not getting the stack trace. Can someone please help?

The call from Go to C switches stacks. The Go runtime doesn't provide
enough information for gdb to be able to unwind past that stack
switch. gdb can show the stack trace of the C function calls, but not
the stack trace of the Go functions.

Ian

[mariappan balraj]

Hi Ian,

Thanks for your reply. This point is clear. I am interested only in getting the complete C call stack only. I could able to get the GO stack by using the delve debugger. 

When I use the GDB, I am getting the following stack. In the stack I am seeing the last C function called which test1(). But I am not seeing test2() and test3() in the stack. In GO code, test3() is called. These details are very important when we want to debug the issues from production. 

I am eagerly waiting for the solution. It really helps others also. Kindly please help on this.

(gdb) thread apply all bt

Thread 3 (Thread 0x7f2d70694740 (LWP 182930)):

#0  runtime.usleep () at /usr/local/go/src/runtime/sys_linux_amd64.s:140

#1  0x0000000000448fd8 in runtime.sighandler (sig=6, info=<optimized out>, ctxt=<optimized out>, gp=0xc0000061a0) at /usr/local/go/src/runtime/signal_unix.go:789

#2  0x00000000004484a5 in runtime.sigtrampgo (sig=6, info=0xc00000fbf0, ctx=0xc00000fac0) at /usr/local/go/src/runtime/signal_unix.go:479

#3  0x000000000045fba6 in runtime.sigtramp () at /usr/local/go/src/runtime/sys_linux_amd64.s:359

#4  <signal handler called>

#5  0x00007f2d7072da7c in pthread_kill () from /lib/x86_64-linux-gnu/libc.so.6

#6  0x00007f2d706d9476 in raise () from /lib/x86_64-linux-gnu/libc.so.6

#7  0x00007f2d706bf7f3 in abort () from /lib/x86_64-linux-gnu/libc.so.6

#8  0x00007f2d706bf71b in ?? () from /lib/x86_64-linux-gnu/libc.so.6

#9  0x00007f2d706d0e96 in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6

#10 0x0000000000462be7 in test1 () at /home/ubuntu/mbalraj/GO/TEST/test.go:10

#11 0x000000000045dce4 in runtime.asmcgocall () at /usr/local/go/src/runtime/asm_amd64.s:844

#12 0x00000000004dd620 in ?? ()

#13 0x0000000000000001 in ?? ()

#14 0x000000c000080a00 in ?? ()

#15 0x0a007ffebe0a3b28 in ?? ()

#16 0x0000000000000001 in ?? ()

#17 0x00000000000000f8 in ?? ()

#18 0x000000c0000061a0 in ?? ()

#19 0x000000c000058ae0 in ?? ()

#20 0x000000000045db29 in runtime.systemstack () at /usr/local/go/src/runtime/asm_amd64.s:492

#21 0x00000000004604e5 in runtime.newproc (fn=0x1) at <autogenerated>:1

#22 0x00000000004c57c0 in runtime[scavenger] ()

#23 0x0000000000000001 in ?? ()

#24 0x000000000045da25 in runtime.mstart () at /usr/local/go/src/runtime/asm_amd64.s:390

#25 0x000000000045d9af in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:354

#26 0x0000000000000001 in ?? ()

#27 0x00007ffebe0a3ca8 in ?? ()

#28 0x0000000000000000 in ?? ()

By using dlv, I am seeing the following GO stack.

(dlv) goroutine 1 bt

0  0x000000000045f85d in runtime.usleep

   at /usr/local/go/src/runtime/sys_linux_amd64.s:140

1  0x000000000045dac0 in runtime.systemstack_switch

   at /usr/local/go/src/runtime/asm_amd64.s:459

2  0x0000000000404c4a in runtime.cgocall

   at /usr/local/go/src/runtime/cgocall.go:168

3  0x0000000000462b45 in main._Cfunc_test3

   at _cgo_gotypes.go:41

4  0x0000000000462b97 in main.main

   at ./test.go:24

5  0x0000000000437458 in runtime.main

   at /usr/local/go/src/runtime/proc.go:250

6  0x000000000045dfe1 in runtime.goexit

   at /usr/local/go/src/runtime/asm_amd64.s:1594

My GDB version is 

gdb --version

GNU gdb (GDB) 12.1

go version go1.19.4 linux/amd64

Best Regards

Mariappan

[Ian Lance Taylor]

On Thu, Jan 5, 2023 at 9:02 PM mariappan balraj
<mariappa...@gmail.com> wrote:
>
> Thanks for your reply. This point is clear. I am interested only in getting the complete C call stack only. I could able to get the GO stack by using the delve debugger.
>
> When I use the GDB, I am getting the following stack. In the stack I am seeing the last C function called which test1(). But I am not seeing test2() and test3() in the stack. In GO code, test3() is called. These details are very important when we want to debug the issues from production.
>
> I am eagerly waiting for the solution. It really helps others also. Kindly please help on this.

Please include plain text as plain text, not in colors with a
background. Plain text is much easier to read in e-mail. Thanks.

I expect that you are only seeing test1 because the C code is compiled
with optimization and all the functions are inlined. Try building
with `CGO_CFLAGS=-g` set in the environment.

Ian

[mariappan balraj]

Hi Ian,

Thanks for your quick reply. I have now tried setting the CGO_CFLAGS to -g. Now I am seeing the following BT. It is reporting "corrupt stack". 

(gdb) bt
#0  runtime.raise () at /usr/local/go/src/runtime/sys_linux_amd64.s:159
#1  0x0000000000443345 in runtime.dieFromSignal (sig=6) at /usr/local/go/src/runtime/signal_unix.go:870
#2  0x00000000004438de in runtime.sigfwdgo (sig=6, info=<optimized out>, ctx=<optimized out>, ~r0=<optimized out>)
    at /usr/local/go/src/runtime/signal_unix.go:1086
#3  0x0000000000442027 in runtime.sigtrampgo (sig=0, info=0x0, ctx=0x4583c1 <runtime.raise+33>) at /usr/local/go/src/runtime/signal_unix.go:432
#4  0x00000000004586a6 in runtime.sigtramp () at /usr/local/go/src/runtime/sys_linux_amd64.s:359
#5  <signal handler called>
#6  runtime.raise () at /usr/local/go/src/runtime/sys_linux_amd64.s:159
#7  0x0000000000443345 in runtime.dieFromSignal (sig=6) at /usr/local/go/src/runtime/signal_unix.go:870
#8  0x0000000000443558 in runtime.crash () at /usr/local/go/src/runtime/signal_unix.go:962
#9  0x000000000042f891 in runtime.fatalthrow.func1 () at /usr/local/go/src/runtime/panic.go:1129
#10 0x000000000042f80c in runtime.fatalthrow (t=<optimized out>) at /usr/local/go/src/runtime/panic.go:1122
#11 0x000000000042f4bd in runtime.throw (s=...) at /usr/local/go/src/runtime/panic.go:1047
#12 0x0000000000443289 in runtime.sigpanic () at /usr/local/go/src/runtime/signal_unix.go:819
#13 0x000000000045abe6 in test1 () at /home/ubuntu/mbalraj/GO/TEST/test.go:9
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

root@nvme-tcp:/home/ubuntu/mbalraj/GO/TEST# go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.4"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/ubuntu/mbalraj/GO/TEST/go.mod"
GOWORK=""
CGO_CFLAGS="-g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1125623398=/tmp/go-build -gno-record-gcc-switches"

Best Regards

Mariappan

[mariappan balraj]

Hi Ian,

When I call the same function from pure C code, I am able to get the complete stack and it is not reporting that the stack is corrupted. I am expecting the same C stack when I use CGO also. Please kindly help with this. 

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000559e2af8813d in test1 () at test.c:6
6   *p = 30;
(gdb) bt
#0  0x0000559e2af8813d in test1 () at test.c:6
#1  0x0000559e2af88153 in test2 () at test.c:11
#2  0x0000559e2af88163 in test3 () at test.c:15
#3  0x0000559e2af88173 in main () at test.c:19

#include <stdio.h>
#include <assert.h>

void test1(void) {
   int *p = (int*)NULL;
   *p = 30;

   //assert(1 == 2);

}

void test2(void) {
    test1();
}

void test3(void) {
    test2();
}

int main(void) {
   test3();
}

Best Regards

Mariappan

[Ian Lance Taylor]

On Thu, Jan 5, 2023 at 10:39 PM mariappan balraj
<mariappa...@gmail.com> wrote:
>
> When I call the same function from pure C code, I am able to get the complete stack and it is not reporting that the stack is corrupted. I am expecting the same C stack when I use CGO also. Please kindly help with this.

C code and Go code run on different stacks. As I tried to explain
earlier, gdb does not have enough information to unwind from a C stack
to a Go stack.

What you are looking for simply doesn't work. Sorry.

Ian

[mariappan balraj]

Hi Ian, 

I am not expecting GO stack. I am interested only in getting C stack. If I want go stack, I can use delve debugger to get it. From GO, using CGO, test3() is called which is calling test2() which is calling test1(). I am expecting only C stack which contains test3(),  test2(), test1(). In this particular case assigning value by using pointer variable which contains NULL(segmentation fault). I am seeing only test1(). After that it is not stack and saying stack corruption. I strongly believe that you can help on this. Please help. 

Best regards

Mariappan

[Ian Lance Taylor]

On Fri, Jan 6, 2023 at 5:28 PM mariappan balraj
<mariappa...@gmail.com> wrote:
>
> I am not expecting GO stack. I am interested only in getting C stack. If I want go stack, I can use delve debugger to get it. From GO, using CGO, test3() is called which is calling test2() which is calling test1(). I am expecting only C stack which contains test3(), test2(), test1(). In this particular case assigning value by using pointer variable which contains NULL(segmentation fault). I am seeing only test1(). After that it is not stack and saying stack corruption. I strongly believe that you can help on this. Please help.

I put your program in foo.go. Then I did:

> CGO_CFLAGS=-g go build foo.go
> gdb ./foo
GNU gdb (Debian 12.1-3) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./foo...
warning: File "/home/iant/go/src/runtime/runtime-gdb.py" auto-loading
has been declined by your `auto-load safe-path' set to
"$debugdir:$datadir/auto-load".
To enable execution of this file add
add-auto-load-safe-path /home/iant/go/src/runtime/runtime-gdb.py
line to your configuration file "/home/iant/.config/gdb/gdbinit".
To completely disable this security protection add
set auto-load safe-path /
line to your configuration file "/home/iant/.config/gdb/gdbinit".
For more information about this security protection see the
--Type <RET> for more, q to quit, c to continue without paging--
"Auto-loading safe path" section in the GDB manual. E.g., run from the shell:
info "(gdb)Auto-loading safe path"
(gdb) r
Starting program: /tmp/x/foo
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7fffd09ea640 (LWP 650585)]
[New Thread 0x7fffcbfff640 (LWP 650586)]
[New Thread 0x7fffcb7fe640 (LWP 650587)]
[New Thread 0x7fffcaffd640 (LWP 650588)]
[New Thread 0x7fffca7fc640 (LWP 650589)]

Thread 1 "foo" received signal SIGSEGV, Segmentation fault.
0x000000000045b01a in test1 () at /home/iant/foo.go:6
6 *p = 30;
(gdb) where
#0 0x000000000045b01a in test1 () at /home/iant/foo.go:6
#1 0x000000000045b02c in test2 () at /home/iant/foo.go:10
#2 0x000000000045b038 in test3 () at /home/iant/foo.go:14
#3 0x000000000045b054 in _cgo_3060c004c901_Cfunc_test3 (v=0xc000064f70)
at /tmp/go-build/cgo-gcc-prolog:49
#4 0x0000000000456c64 in runtime.asmcgocall ()
at /home/iant/go/src/runtime/asm_amd64.s:848
#5 0x00000000004e3460 in ?? ()
#6 0x0000000000000001 in ?? ()
#7 0x000000c000080500 in ?? ()
#8 0x00007fffffffe458 in ?? ()
#9 0x0000000000439225 in runtime.malg.func1 ()
at /home/iant/go/src/runtime/proc.go:4227
#10 0x0000000000456aa9 in runtime.systemstack ()
at /home/iant/go/src/runtime/asm_amd64.s:496
#11 0x00000000004596a5 in runtime.newproc (fn=0x1) at <autogenerated>:1
#12 0x00000000004cc720 in runtime[scavenger] ()

#13 0x0000000000000001 in ?? ()

#14 0x00000000004569a5 in runtime.mstart ()
at /home/iant/go/src/runtime/asm_amd64.s:394
#15 0x000000000045692f in runtime.rt0_go ()
at /home/iant/go/src/runtime/asm_amd64.s:358

#16 0x0000000000000001 in ?? ()

--Type <RET> for more, q to quit, c to continue without paging--q
Quit



So when I try it, I do see the full C stack at the point where the
signal occurs.

In your backtrace earlier you are trying to see the stack after the
signal is already being handled by the Go signal handler. I don't
know why that would work.

Ian

[mariappan balraj]

Hi Ian,

Thanks for your active help. When I run the program by using gdb, I am getting the complete stack. No issue. The issue is there when we debug core dump. Could you kindly please check whether you are seeing the same behavior with core dump? 

Best Regards

Mariappan

[Ian Lance Taylor]

On Fri, Jan 6, 2023, 5:57 PM mariappan balraj <mariappa...@gmail.com> wrote:

Hi Ian,

Thanks for your active help. When I run the program by using gdb, I am getting the complete stack. No issue. The issue is there when we debug core dump. Could you kindly please check whether you are seeing the same behavior with core dump? 

Oh, right, sorry, I forgot about the core dump part.  I don't know if there is a way to make that better, given the three different stacks involved.  I'm surprised that it works as well as it does.  A pure C program that doesn't use sigaltstack only has a single stack, so it's a much simpler case.

Ian

[mariappan balraj]

Hi Ian,

Thanks for your continuous support. GOLANG supports CGO to invoke C functions. When it is supported, the important thing is, it should provide better debugging support when there is any issue. In customer sites, it is not possible to run applications with GDB. Customers only provide core dump and logs. With the provided information, we should be able to debug the issue. It may not be possible to reproduce all the issues in the development environment to debug the issue. 

When we run the application with GDB, we are getting stack trace. Then the same thing should be possible with core dump also. 

I have tried with CGO symbolizer from https://github.com/ianlancetaylor/cgosymbolizer. I am getting the following output. This is useful. But I want to dump the C variables (local and global) to debug the issue. This is very critical when we want to debug some issues. 

What should I do now? How to proceed further? If possible, please provide your help with this.

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x463926]

runtime stack:
runtime.throw({0x49046b?, 0x0?})
/usr/local/go/src/runtime/panic.go:1047 +0x5d fp=0x7ffca8644230 sp=0x7ffca8644200 pc=0x43243d
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:819 +0x369 fp=0x7ffca8644280 sp=0x7ffca8644230 pc=0x446569

goroutine 1 [syscall]:
test1
/home/ubuntu/mbalraj/GO/TEST/test.go:9 pc=0x463926
test2
/home/ubuntu/mbalraj/GO/TEST/test.go:14 pc=0x46393b
test3
/home/ubuntu/mbalraj/GO/TEST/test.go:18 pc=0x46394b
_cgo_64d258852278_Cfunc_test3
/tmp/go-build/cgo-gcc-prolog:49 pc=0x46396b
runtime.asmcgocall
/usr/local/go/src/runtime/asm_amd64.s:844 pc=0x45c443
runtime.cgocall(0x46394f, 0xc000058f70)
/usr/local/go/src/runtime/cgocall.go:158 +0x5c fp=0xc000058f48 sp=0xc000058f10 pc=0x40579c
main._Cfunc_test3()
_cgo_gotypes.go:41 +0x45 fp=0xc000058f70 sp=0xc000058f48 pc=0x463885
main.main()
/home/ubuntu/mbalraj/GO/TEST/test.go:26 +0x17 fp=0xc000058f80 sp=0xc000058f70 pc=0x4638b7
runtime.main()
/usr/local/go/src/runtime/proc.go:250 +0x212 fp=0xc000058fe0 sp=0xc000058f80 pc=0x434c92
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc000058fe8 sp=0xc000058fe0 pc=0x45c761

Best Regards

Mariappan

[Ian Lance Taylor]

On Fri, Jan 6, 2023 at 9:01 PM mariappan balraj
<mariappa...@gmail.com> wrote:
>
> Thanks for your continuous support. GOLANG supports CGO to invoke C functions. When it is supported, the important thing is, it should provide better debugging support when there is any issue. In customer sites, it is not possible to run applications with GDB. Customers only provide core dump and logs. With the provided information, we should be able to debug the issue. It may not be possible to reproduce all the issues in the development environment to debug the issue.
>
> When we run the application with GDB, we are getting stack trace. Then the same thing should be possible with core dump also.
>
> I have tried with CGO symbolizer from https://github.com/ianlancetaylor/cgosymbolizer. I am getting the following output. This is useful. But I want to dump the C variables (local and global) to debug the issue. This is very critical when we want to debug some issues.
>
> What should I do now? How to proceed further? If possible, please provide your help with this.

I'm sorry, I don't have any useful suggestions. It's possible in
principle to unwind the stack yourself by looking carefully at the
instructions that will be executed and the PC and SP registers, and
then to look at the instructions to figure out where variables are
stored, but it's hard and it's easy to make a mistake.

Ian

[mariappan balraj]

Hi Ian,

Thanks for all your replies. It really shows that you have tried to give your best all the time. I need some direction to get a permanent solution for this. Is it possible to get help from the core google GO team? How to escalate this issue and get the fix? Please give me directions. So that I can try best from my side.

Best Regards

Mariappan

[Ian Lance Taylor]

On Sun, Jan 8, 2023, 9:33 PM mariappan balraj <mariappa...@gmail.com> wrote:

Hi Ian,

Thanks for all your replies. It really shows that you have tried to give your best all the time. I need some direction to get a permanent solution for this. Is it possible to get help from the core google GO team? How to escalate this issue and get the fix? Please give me directions. So that I can try best from my side.

I'm on the core Google Go team myself.

The next step is to file a bug report at https://go.dev/issue, with exact details for how to reproduce the problem.  But I don't want to mislead you: it's unlikely that anybody on the core Go team is going to fix this.  That said, Go is an open source project, and filing a bug report is the right step to encourage someone to fix the problem.

It's also worth taking a step back and describing the real problem.  Using gdb to get a stack trace from a core dump is a technique, it's not a solution.  Perhaps there are other techniques.

Ian

[mariappan balraj]

Hi Ian,

Thanks. I will try this. When a process is crashed because of a SEGMENTATION fault, it can be debugged by identifying the stack trace from the core dump. Is there any other technique to debug this issue? Can you please help if any other technique is there?

Best Regards

Mariappan

[Tamás Gulácsi]

I'd create a separate C program, that can be communicated with RPC style - (https://github.com/protobuf-c/protobuf-c-rpc may solve it easily),

and have the Go program start it like https://github.com/hashicorp/go-plugin .

That way the stack traces and core dumps are separate, each program is easier to debug,

the C error does not take down the Go program, which can restart the plugin ...

[mariappan balraj]

Hi Tamas,

Using RPC will not scale and it needs lot of effort. So our only choice is CGO.

Best Regards

Mariappan