Got unknown stack while capturing go binary exit point with bpftrace

[Yeqowcc]

Hi there,

I want to practice bpftrace tracing go program exit point, such as, runtime.gopanic and os.Exit, but I'm confused with the result (there's different stack information between runtime.panic and syscall.Exit/os.Exit):

[root@localhost trace-go-func]# bpftrace -c "./binary -mode exit -code 2" 

trace.bt Attaching 2 probes... 

mode=exit, ecode=2 

syscall.Exit(2) called with stack: 

 458080 0x458080 ([unknown]) 

 487f9e 0x487f9e ([unknown]) 

 487ee6 0x487ee6 ([unknown]) 

 433192 0x433192 ([unknown]) 

 45ba81 0x45ba81 ([unknown]) 

 [root@localhost trace-go-func]# bpftrace -c "./binary -mode panic" trace.bt 

Attaching 2 probes... 

mode=panic, ecode=0 

panic called with stack 

 430280 runtime.gopanic+0 (/root/projects/trace-go-func/binary) 

 487f05 main.main+293 (/root/projects/trace-go-func/binary) 

 433192 runtime.main+530 (/root/projects/trace-go-func/binary) 

 45ba81 runtime.goexit.abi0+1 (/root/projects/trace-go-func/binary) 

panic: doPanic 

goroutine 1 [running]: 

main.doPanic() /root/projects/trace-go-func/main.go:31 +0x27 

main.main() /root/projects/trace-go-func/main.go:22 +0x125

as you see, panic works well, but os.Exit could not get stack correctly. I have found symbols with addresses in nm like this, for x in 458080 487f9e 487ee6; do nm ./binary | grep $x; done, but only 458080 syscall.Exit was got. 

I'm confused now, why bpftrace could get the address and symbols while runtime.gopanic but os.Exit, and even could not be matched in nm?

main.go looks like:

package main 

import ( "flag" "fmt" "os" ) 

var ( 

 mode = flag.String("mode", "exit", "choose mode to run and test") 

 code = flag.Int("code", 0, "os.Exit code") 

) 

func main() { 

 flag.Parse() 

 fmt.Printf("mode=%s, ecode=%d\n", *mode, *code) 

// defer fmt.Println("main quit") 

switch *mode { 

case "panic": doPanic() 

case "exit": doOSExit(*code) 

 } 

} 

//go:noinline 

func doPanic() { 

// fmt.Println("panic calling") 

panic("doPanic") 

} 

//go:noinline func doOSExit(code int) { 

// fmt.Println("os.Exit calling") 

 os.Exit(code)  

}

trace.bt looks like:

uprobe:./binary:runtime.gopanic 

{ 

 printf("panic called with stack %s", ustack(perf)); 

} 

uprobe:./binary:syscall.Exit { 

 printf("syscall.Exit(%d) called with stack: %s", sarg0, ustack(perf)); 

}

Enviroments:

OS: centos7
Kernel: 5.4.188-1.el7.elrepo.x86_64
bpftrace: v0.13.0
go: 1.18

the following link maybe look better

1. linux - got unknown stack while capturing go binary exit point with bpftrace - Stack Overflow