Runtime panic

[Stavros Filargyropoulos]

Hi,

On an embedded 16 core arm device we have been observing infrequent runtime crashes, with different signatures every time.

One examples is this:

unexpected fault address 0x6b6fc6da
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x6b6fc6da pc=0x7f72c]

goroutine 1 [running, locked to thread]:
runtime.throw(0xc3074, 0x5)
    /usr/local/go/src/runtime/panic.go:1116 +0x54 fp=0x4000072c20 sp=0x4000072bf0 pc=0x3dcc4
runtime.sigpanic()
    /usr/local/go/src/runtime/signal_unix.go:749 +0x3b8 fp=0x4000072c50 sp=0x4000072c20 pc=0x51b48
reflect.(*rtype).Method(0xc55fe, 0x14, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /usr/local/go/src/reflect/type.go:809 +0xcc fp=0x4000072de0 sp=0x4000072c60 pc=0x7f72c
runtime: unexpected return pc for runtime.doInit called from 0x4000098060
stack: frame={sp:0x4000072de0, fp:0x4000072e20} stack=[0x4000072000,0x4000073000)
0000004000072ce0:  000000400009a4e0  000000400009c340
0000004000072cf0:  0000000000000006  000000400009c3a8
0000004000072d00:  000000400009c3f8  000000400009c410
0000004000072d10:  0000000000000002  000000400009c438
0000004000072d20:  000000400009c4a8  0000004000072d58
0000004000072d30:  000000000001fb14 <runtime.mapassign_faststr+804>  00000000000ae3e0
0000004000072d40:  0000004000098060  0000000000000000
0000004000072d50:  00000040000a4d00  0000004000072dc8
0000004000072d60:  000000000007d790 <unicode.init+16592>  00000000000ae3e0
0000004000072d70:  0000004000098060  0000000000000001
0000004000072d80:  000000400009a4e0  0000000000002001
0000004000072d90:  000000400009806e  0000004000072dc8
0000004000072da0:  000000000007d4c0 <unicode.init+15872>  6e97779bde52831f
0000004000072db0:  0000000000000001  0000000000000000
0000004000072dc0:  000000400009c410  0000004000072e28
0000004000072dd0:  000000000004d02c <runtime.doInit+156>  00000000000ae3e0
0000004000072de0: <0000004000098060  00000000000c55fe
0000004000072df0:  0000000000000014  0000000000000000
0000004000072e00:  0000000000000000  0000000000000000
0000004000072e10:  0000000000000000  0000000000000000
0000004000072e20: >0000000000000000  0000000000000000
0000004000072e30:  0000000000000000  0000000000000000

0000004000072e40:  0000000000000000  0000004000072e78
0000004000072e50:  0000000000000000  0000000000000000
0000004000072e60:  0000000000150e98  0000004000072ea8
0000004000072e70:  000000000004cfe8 <runtime.doInit+88>  0000000000150e80
0000004000072e80:  000000000004d02c <runtime.doInit+156>  00000000000c45db
0000004000072e90:  000000000000000e  0000000000000002
0000004000072ea0:  0000004000096010  0000004000072ee8
0000004000072eb0:  000000000004cfe8 <runtime.doInit+88>  0000000000151ec0
0000004000072ec0:  0000000000000060  0000007f9bd197d0
0000004000072ed0:  0000000000000000  0000000000000000
0000004000072ee0:  00000000001518e8  0000004000072f28
0000004000072ef0:  000000000004cfe8 <runtime.doInit+88>  0000000000151500
0000004000072f00:  0000000000000000  000000400004c738
0000004000072f10:  0000000000025004 <runtime.gcenable+148>  0000000000000002
runtime.doInit(0x0)
    /usr/local/go/src/runtime/proc.go:5652 +0x9c fp=0x4000072e20 sp=0x4000072de0 pc=0x4d02c

goroutine 2 [force gc (idle)]:
runtime.gopark(0xcaff8, 0x165dd0, 0x1411, 0x1)
    /usr/local/go/src/runtime/proc.go:306 +0xd0 fp=0x400004cfa0 sp=0x400004cf80 pc=0x40540
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:312
runtime.forcegchelper()
    /usr/local/go/src/runtime/proc.go:255 +0xc4 fp=0x400004cfd0 sp=0x400004cfa0 pc=0x403f4
runtime.goexit()
    /usr/local/go/src/runtime/asm_arm64.s:1136 +0x4 fp=0x400004cfd0 sp=0x400004cfd0 pc=0x6b8f4
created by runtime.init.5
    /usr/local/go/src/runtime/proc.go:243 +0x30

goroutine 3 [GC sweep wait]:
runtime.gopark(0xcaff8, 0x165ee0, 0x140c, 0x1)
    /usr/local/go/src/runtime/proc.go:306 +0xd0 fp=0x400004d7a0 sp=0x400004d780 pc=0x40540
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:312
runtime.bgsweep(0x400006a000)
    /usr/local/go/src/runtime/mgcsweep.go:163 +0xb0 fp=0x400004d7d0 sp=0x400004d7a0 pc=0x2e880
runtime.goexit()
    /usr/local/go/src/runtime/asm_arm64.s:1136 +0x4 fp=0x400004d7d0 sp=0x400004d7d0 pc=0x6b8f4
created by runtime.gcenable
    /usr/local/go/src/runtime/mgc.go:217 +0x54

goroutine 4 [GC scavenge wait]:
runtime.gopark(0xcaff8, 0x165f80, 0x140d, 0x1)
    /usr/local/go/src/runtime/proc.go:306 +0xd0 fp=0x400004df70 sp=0x400004df50 pc=0x40540
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:312
runtime.bgscavenge(0x400006a000)
    /usr/local/go/src/runtime/mgcscavenge.go:265 +0xec fp=0x400004dfd0 sp=0x400004df70 pc=0x2cd4c
runtime.goexit()

    /usr/local/go/src/runtime/asm_arm64.s:1136 +0x4 fp=0x400004dfd0 sp=0x400004dfd0 pc=0x6b8f4
created by runtime.gcenable
    /usr/local/go/src/runtime/mgc.go:218 +0x74

The source code for the app is at: https://github.com/stavrospen/gomemverify

This is with go version go1.15.11 linux/amd64, cross compiling for arm64. CGO disabled.

We also use upx version 3.95

The background is that we have been seeing our go application crash, and were suspecting some external entity corrupting the memory, so created this small utility to try to catch memory corruption. We never managed to catch any corruption in the heap, but we keep seeing these random crashes with what seems to be stack corruption.

I should note here, that we also have C++ application running side by side, but we have never seen any similar crash there.

Was wondering if someone has any idea on how to debug this further, or what Go tools we can use to try to capture the possible corruption. We have tried 'GODEBUG="efence=1"' but didn't seem to catch anything.

[Ian Lance Taylor]

On Tue, Mar 15, 2022 at 9:09 PM Stavros Filargyropoulos
<sta...@pensando.io> wrote:
>
> On an embedded 16 core arm device we have been observing infrequent runtime crashes, with different signatures every time.
>
> One examples is this:
>
> unexpected fault address 0x6b6fc6da
> fatal error: fault
> [signal SIGSEGV: segmentation violation code=0x1 addr=0x6b6fc6da pc=0x7f72c]
>
> goroutine 1 [running, locked to thread]:
> runtime.throw(0xc3074, 0x5)
> /usr/local/go/src/runtime/panic.go:1116 +0x54 fp=0x4000072c20 sp=0x4000072bf0 pc=0x3dcc4
> runtime.sigpanic()
> /usr/local/go/src/runtime/signal_unix.go:749 +0x3b8 fp=0x4000072c50 sp=0x4000072c20 pc=0x51b48
> reflect.(*rtype).Method(0xc55fe, 0x14, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
> /usr/local/go/src/reflect/type.go:809 +0xcc fp=0x4000072de0 sp=0x4000072c60 pc=0x7f72c
> runtime: unexpected return pc for runtime.doInit called from 0x4000098060

...

> This is with go version go1.15.11 linux/amd64, cross compiling for arm64. CGO disabled.

There have been some fixes in this area. Is it possible for you to
try the new 1.18 release to see if the problem still happens? (Note
that we no longer support 1.15.)

> We also use upx version 3.95

There have been some reported problems using upx with Go binaries. Is
it possible for you to try not using upx to see if the problem still
happens?

Ian

[Stavros Filargyropoulos]

Thanks Ian! Removing UPX solved the problem.