How to debug signal 0xb panic

[Hoping White]

Hi, all

I have a cgo related program, and it crashes while a 0xb signal received.

For a c/cpp program, I can get a core dump file and gdb it to get the segment violation location.

But for golang, I don’t know how to do that. I can’t find a core dump file and only get a panic stack trace as following:

[signal 0xb code=0x1 addr=0x8 pc=0x4201c75]

runtime stack:

runtime.throw(0x4340760, 0x2a)

/Users/hoping/Programs/Golang/go/src/runtime/panic.go:527 +0x90

runtime.sigpanic()

/Users/hoping/Programs/Golang/go/src/runtime/sigpanic_unix.go:12 +0x5a

goroutine 44 [syscall, locked to thread]:

runtime.cgocall(0x41feef0, 0xc82004ded0, 0xc800000000)

/Users/hoping/Programs/Golang/go/src/runtime/cgocall.go:120 +0x11b fp=0xc82004de38 sp=0xc82004de08

babeltime.com/ncmc._Cfunc_cmc_traverse(0xc820080070, 0xc82006e690, 0x64, 0xc86023e000, 0x100000)

babeltime.com/ncmc/_obj/_cgo_gotypes.go:286 +0x40 fp=0xc82004ded0 sp=0xc82004de38

babeltime.com/ncmc.Traverse(0x4361c20, 0xc82006e690, 0x64, 0x64, 0xc86023e000, 0x100000, 0x100000)

/Users/hoping/Programs/Golang/workspace/ARPGServer/src/babeltime.com/ncmc/ncmc.go:141 +0xf8 fp=0xc82004df28 sp=0xc82004ded0

babeltime.com/nads.(*iDataStore).doTraverse(0xc820072240)

/Users/hoping/Programs/Golang/workspace/ARPGServer/src/babeltime.com/nads/traverse.go:66 +0x11f fp=0xc82004df98 sp=0xc82004df28

runtime.goexit()

/Users/hoping/Programs/Golang/go/src/runtime/asm_amd64.s:1721 +0x1 fp=0xc82004dfa0 sp=0xc82004df98

created by babeltime.com/nads.(*iDataStore).startTraverse

/Users/hoping/Programs/Golang/workspace/ARPGServer/src/babeltime.com/nads/traverse.go:58 +0xc1

And many other goroutine trace.

I use go version go1.5.2 darwin/amd64

[Konstantin Khomoutov]

[...]

Your /Users suggests you're on Mac OS X. Is that correct?

Are you sure 0xb (11) is SIGSEGV there as well? (It is, on Linux.)

In either case, with Go 1.5 you can run your program while having

GOTRACEBACK=crash

set in its environment to produce a core dump upon the crash and then
try inspecting it in GDB. Make sure your `ulimit` settings allow
generating core dumps as well. See [1] for more.

As you can see, the crash happens either in cgo-generated core or on
the C side (which is sadly the most typical, if not sole, reason for
such crashes in Go programs). So I'd first checked and re-checked
whether you're playing well with your C library. This might save you
from inspecting the dump in the debugger.

[2] has a nice summary (and further pointers) of the rules Go code must
play by when interacting with C code under the promises Go's (now
concurrent) GC makes.

1. http://dave.cheney.net/2015/11/29/a-whirlwind-tour-of-gos-runtime-environment-variables
2. https://github.com/golang/go/issues/12416

[Hoping White]

Hi

I have tried GOTRACEBACK=crash on linux, and got a core dump file

#0  runtime.raise () at /home/xingli/programs/golang/go/src/runtime/sys_linux_amd64.s:110

#1  0x0000000000441627 in runtime.crash () at /home/xingli/programs/golang/go/src/runtime/signal1_unix.go:195

#2  0x000000000042f296 in runtime.dopanic_m (gp=0xa4a0e0, pc=4385920, sp=140734441199696)

    at /home/xingli/programs/golang/go/src/runtime/panic1.go:119

#3  0x00000000004550d2 in runtime.dopanic.func1 () at /home/xingli/programs/golang/go/src/runtime/panic.go:514

#4  0x000000000045efcb in runtime.systemstack () at /home/xingli/programs/golang/go/src/runtime/asm_amd64.s:278

#5  0x000000000042ebe1 in runtime.dopanic (unused=0) at /home/xingli/programs/golang/go/src/runtime/panic.go:515

#6  0x000000000042ec80 in runtime.throw (s=...) at /home/xingli/programs/golang/go/src/runtime/panic.go:527

#7  0x0000000000442efa in runtime.sigpanic () at /home/xingli/programs/golang/go/src/runtime/sigpanic_unix.go:12

#8  0x0000000000411427 in runtime.unlock (l=0xc82001c000) at /home/xingli/programs/golang/go/src/runtime/lock_futex.go:103

#9  0x000000000043e495 in runtime.(*guintptr).cas (gp=0xc82001cdc0, old=859530632896, new=0, ~r2=8)

    at /home/xingli/programs/golang/go/src/runtime/runtime2.go:133

#10 0x000000000043bc16 in runtime.runqput (_p_=0x460ac0, gp=0xc820049e48, next=90) at /home/xingli/programs/golang/go/src/runtime/proc1.go:3437

#11 0x000000000070f310 in go.string.* ()

#12 0x0000000000460ac0 in runtime.asmcgocall () at /home/xingli/programs/golang/go/src/runtime/asm_amd64.s:692

#13 0x000000c820049e48 in ?? ()

#14 0x0000000000405e5a in runtime.cgocall (fn=0xa4a120, arg=0x7fff4a6019b0, ~r2=10789088)

    at /home/xingli/programs/golang/go/src/runtime/cgocall.go:107

#15 0x0000000000433a60 in runtime.startTheWorldWithSema () at /home/xingli/programs/golang/go/src/runtime/proc1.go:603

#16 0x00007fff4a601ac8 in ?? ()

#17 0x0000000000000001 in ?? ()

#18 0x00007fff4a601ac8 in ?? ()

#19 0x0000000000000000 in ?? ()

There is no my go or cpp file, what does this mean?

[Ian Lance Taylor]

The most likely explanation is that your C code has corrupted memory
belonging to the Go runtime.

If you are using Go tip and clang, try building with `go install -msan`.

Ian

[Hoping White]

Hi, Ian

I interchange data between go and cpp using the following pattern

func getByteSlicePointer(data []byte) unsafe.Pointer {

if data == nil {

return unsafe.Pointer(nil)

}

header := (*reflect.SliceHeader)(unsafe.Pointer(&data))

return unsafe.Pointer(header.Data)

}

func setByteSlicePointer(data *[]byte, buffer unsafe.Pointer, size uint32) {

header := (*reflect.SliceHeader)(unsafe.Pointer(data))

header.Len = int(size)

header.Cap = int(size)

header.Data = uintptr(buffer)

}

func getStringPointer(s *string) unsafe.Pointer {

if s == nil {

return unsafe.Pointer(nil)

}

header := (*reflect.StringHeader)(unsafe.Pointer(s))

return unsafe.Pointer(header.Data)

}

And I have tested this package contains cpp code along, and it’s ok. I didn’t found any fault, and it’s stable. The problem occurred when I use this package in another package.

It always failed in that package, and it seems runtime have been ruined. Any ideas?

By the way, I have tried -msan with no luck. It seems my go version does not support msan.

Can you be more specific about this?

Thanks.

[Konstantin Khomoutov]

On Thu, 24 Dec 2015 13:26:51 +0800
Hoping White <baiha...@gmail.com> wrote:

> Hi, Ian
>
> I interchange data between go and cpp using the following pattern
>
>
> func getByteSlicePointer(data []byte) unsafe.Pointer {
> if data == nil {
> return unsafe.Pointer(nil)
> }
> header := (*reflect.SliceHeader)(unsafe.Pointer(&data))
> return unsafe.Pointer(header.Data)
> }
>
> func setByteSlicePointer(data *[]byte, buffer unsafe.Pointer, size
> uint32) { header := (*reflect.SliceHeader)(unsafe.Pointer(data))
> header.Len = int(size)
> header.Cap = int(size)
> header.Data = uintptr(buffer)
> }
>
>
> func getStringPointer(s *string) unsafe.Pointer {
> if s == nil {
> return unsafe.Pointer(nil)
> }
> header := (*reflect.StringHeader)(unsafe.Pointer(s))
> return unsafe.Pointer(header.Data)
> }

Don't use `reflect.*Header` -- the Go authors actually regret exposing
this implementation detail. Use cgo-provided facilities and a
type-conversion trick to convert C byte-arrays to Go's byte slices.
Please read [1] for more info.

On a side note, your examples miss context. Are you positive that once
you passed the data from your C side to Go it stays alive (that is, not
free()d or realloc()ed during its entire lifetime)? The same goes for
Go: once you applied your getByteSlicePointer() to a Go value of type
[]byte and passed the resuting raw pointer to the C side, are you sure
your Go variable does not go out of scope and loses a last reference to
it (and gets garbage collected)? Note that a pointer passed to a C
side it not counted as a reference to a variable: it only is until the
call into C code is in proress; once it's over, the C side must not use
this pointer anymore because the GC on the Go side might have moved the
memory pointed to by that pointer or reclaimed it.

If you need a buffer truly shared by C and Go sides, you either should
obtain it through C.malloc() (and then guarantee the C side won't call
realloc() on that memory) or use a custom allocator on the Go side
(say, allocate a memory pool by using syscall.Mmap with MAP_ANONYMOUS
and then allocate your buffers there; pointers to these buffers may be
freely shared between C and Go sides).

> By the way, I have tried -msan with no luck. It seems my go version
> does not support msan. Can you be more specific about this?

In order for -msan to work, you must be on linux/amd64 and use clang (a
C compiler from the LLVM suite), not gcc, to build your cgo code.

Supposedly you need to export the CC (or CXX if your C side is C++ in
fact) environment variable set to "clang". You might also need to
massage the compiler flags a bit (I dunno). Consider reading [2] and
study the output of `go env` to get a better idea.

[3] is what Go uses when building with -msan and clang.

1. https://github.com/golang/go/wiki/cgo
2. https://golang.org/cmd/cgo/
3. http://clang.llvm.org/docs/MemorySanitizer.html

[Hoping White]

Hi, all

Thanks for all the reply. Finally I have found what’s wrong with my code.
I made a stupid mistake in my test code. I create a buffer as a local variable, and send that to the c library.
But I check and check my logic code and found nothing. And that’s why my ego library test is all good.

Any way, thanks all.