Panic with runtime cgo and go

[webus...@gmail.com]

This program runs for about an hour and consumes about 1Gig of memory. I was getting a lot of these errors, but they were significantly reduced when I reduced the golang memory usage.  (by reducing the in memory state that I was keeping)

Any idea on how I can go about debugging this?

----------------------------------

fatal error: unexpected signal during runtime execution

[signal SIGSEGV: segmentation violation code=0x1 addr=0x9 pc=0x427b64]

runtime stack:

runtime.throw(0x687143, 0x2a)

/go/src/runtime/panic.go:605 +0x97

runtime.sigpanic()

/go/src/runtime/signal_unix.go:351 +0x2bc

runtime.newArenaMayUnlock(0xad3800)

/go/src/runtime/mheap.go:1683 +0x34

runtime.newMarkBits(0x55, 0x55)

/go/src/runtime/mheap.go:1598 +0xb5

runtime.(*mspan).sweep(0x7fd0c05133a0, 0xc420000f00, 0xc420000f00)

/go/src/runtime/mgcsweep.go:311 +0x4ed

runtime.sweepone(0x4355e8)

/go/src/runtime/mgcsweep.go:113 +0x126

runtime.gosweepone.func1()

/go/src/runtime/mgcsweep.go:137 +0x2d

runtime.systemstack(0xc4204964b8)

/go/src/runtime/asm_amd64.s:344 +0x72

runtime.mstart()

/go/src/runtime/proc.go:1125

goroutine 3 [running]:

runtime.systemstack_switch()

/go/src/runtime/asm_amd64.s:298 fp=0xc42003ef70 sp=0xc42003ef68 pc=0x458730

runtime.gosweepone(0x0)

/go/src/runtime/mgcsweep.go:136 +0x4c fp=0xc42003efa0 sp=0xc42003ef70 pc=0x4223cc

runtime.bgsweep(0xc420070000)

/go/src/runtime/mgcsweep.go:55 +0xc1 fp=0xc42003efd8 sp=0xc42003efa0 pc=0x421ee1

runtime.goexit()

/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc42003efe0 sp=0xc42003efd8 pc=0x45b351

created by runtime.gcenable

/go/src/runtime/mgc.go:216 +0x5a

goroutine 6 [chan receive, 9 minutes]:

database/sql.(*DB).connectionOpener(0xc4200c5040)

/go/src/database/sql/sql.go:871 +0x55

created by database/sql.Open

/go/src/database/sql/sql.go:609 +0x1f0

goroutine 17 [running, locked to thread]:

goroutine running on other thread; stack unavailable

goroutine 5 [chan receive, 2 minutes]:

github.com/golang/glog.(*loggingT).flushDaemon(0xad3b80)

/src/github.com/golang/glog/glog.go:879 +0xa1

created by github.com/golang/glog.init.0

/src/github.com/golang/glog/glog.go:410 +0x205

[webus...@gmail.com]

Some other errors that I've gotten from the the same program. 

 

fatal error: unexpected signal during runtime execution

runtime: pointer 0xc43d78cba0 to unallocated span idx=0xebc6 span.base()=0xbdf102 span.limit=0xbdf10200000000 span.state=0

runtime: found in object at *(0xc444740000+0x1d8)

object=0xc444740000 k=0x62223a0 s.base()=0xc444740000 s.limit=0xc444742000 s.spanclass=86 s.elemsize=4096 s.state=_MSpanInUse

 ...

 *(object+464) = 0xc43d6e1380

 *(object+472) = 0xc43d78cba0 <==

 *(object+480) = 0xc43d81fec0

 *(object+1016) = 0xc442693c80

 ...

fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x48 pc=0x676461]

runtime: s.allocCount= 75 s.nelems= 85

fatal error: s.allocCount != s.nelems && freeIndex == s.nelems

goroutine 17 [running, locked to thread]:

runtime.throw(0x688af5, 0x31)

/go/src/runtime/panic.go:605 +0x97 fp=0xc420052ba8 sp=0xc420052b88 pc=0x42d7b7

runtime.(*mcache).nextFree(0x7f11eb9fb000, 0xf, 0xc421bdf800, 0x7f11eb8e0700, 0xc420633b01)

/go/src/runtime/malloc.go:554 +0x266 fp=0xc420052c00 sp=0xc420052ba8 pc=0x4132d6

runtime.mallocgc(0x60, 0x967ae0, 0xab7401, 0xc4206373e0)

/go/src/runtime/malloc.go:711 +0x70c fp=0xc420052ca8 sp=0xc420052c00 pc=0x4139fc

runtime.newobject(0x967ae0, 0xc421bdfec0)

/go/src/runtime/malloc.go:840 +0x3a fp=0xc420052cd8 sp=0xc420052ca8 pc=0x413d3a

sv/common/stats.(*BookSnapshotForDay).TakeSnapshot(...)

/sv/src/nxclinux/historicalclose/go/normalize.go:14

main.(*Normalize).UpdateTime(0xc4200b0bd0, 0x2130c25)

/sv/src/nxclinux/historicalclose/go/normalize.go:247 +0x1c5 fp=0xc420052db0 sp=0xc420052cd8 pc=0x676475

main.(*Normalize).NxcQuoteMMFull(0xc4200b0bd0, 0x2130c25, 0x18000377e3, 0x4e, 0x67, 0x4078080000000000, 0x4078100000000000, 0xc420000600)

/sv/src/nxclinux/historicalclose/go/normalize.go:345 +0x1d7 fp=0xc420052df8 sp=0xc420052db0 pc=0x677807

main.GoNxcQuoteMMFull(0x2130c25, 0x18000377e3, 0x670000004e, 0x4078080000000000, 0x4078100000000000, 0x0)

/sv/src/nxclinux/historicalclose/go/main.go:53 +0x82 fp=0xc420052e48 sp=0xc420052df8 pc=0x6743c2

main._cgoexpwrap_73163f07de6b_GoNxcQuoteMMFull(0x2130c25, 0x18000377e3, 0x670000004e, 0x4078080000000000, 0x4078100000000000, 0x0)

_/sv/src/nxclinux/historicalclose/go/_obj/_cgo_gotypes.go:131 +0x70 fp=0xc420052e88 sp=0xc420052e48 pc=0x674000

runtime.call64(0x0, 0x7ffe83f64eb8, 0x7ffe83f64f70, 0x30)

/go/src/runtime/asm_amd64.s:510 +0x3d fp=0xc420052ed8 sp=0xc420052e88 pc=0x458b6d

runtime.cgocallbackg1(0x0)

/go/src/runtime/cgocall.go:305 +0x1a0 fp=0xc420052f58 sp=0xc420052ed8 pc=0x404d10

runtime.cgocallbackg(0x0)

/go/src/runtime/cgocall.go:187 +0x95 fp=0xc420052fc0 sp=0xc420052f58 pc=0x404ad5

runtime.cgocallback_gofunc(0x0, 0x0, 0x0, 0x0)

/go/src/runtime/asm_amd64.s:762 +0x9a fp=0xc420052fe0 sp=0xc420052fc0 pc=0x45a0fa

runtime.goexit()

/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420052fe8 sp=0xc420052fe0 pc=0x45b351

goroutine 6 [chan receive, 5 minutes]:

database/sql.(*DB).connectionOpener(0xc4200c5180)

/go/src/database/sql/sql.go:871 +0x55

created by database/sql.Open

/go/src/database/sql/sql.go:609 +0x1f0

[Ian Lance Taylor]

On Fri, Sep 15, 2017 at 7:30 AM, <webus...@gmail.com> wrote:
>
> This program runs for about an hour and consumes about 1Gig of memory. I was
> getting a lot of these errors, but they were significantly reduced when I
> reduced the golang memory usage. (by reducing the in memory state that I
> was keeping)
>
> Any idea on how I can go about debugging this?

You neglected to tell us which version of Go you are running. If it's
1.9 then it looks like some form of memory corruption. Have you tried
running the program under the race detector? Do you use the unsafe
package, or cgo? If you use cgo, try running with GODEBUG=cgocheck=2.
Either way, try running with GODEBUG=gccheckmark=1. With luck those
may help you focus in on where the problem occurs.

Ian

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[webus...@gmail.com]

Thanks and Sorry for not providing more details. I'm on 1.9. No unsafe package is used. Main program is a c program which calls into go functions where some analytics are performed. 

All calls to the golang use only strings and ints/floats/char. No structs are passed. In Go all strings are converted to go strings using C.GoString, while int/doubles/chars are converted to values using int64(x) float64(x) uint8(x), etc..

Ok, will run with the debug commands you provided.