checkptr reports error in lfstack_64bit.go

[David Pacheco]

Hello,

I’m trying to track down an issue reported by the Go memory allocator.  Most of the failure modes look like this one:

https://github.com/golang/go/issues/53289#issuecomment-1292744833

but some of them look like:

https://github.com/golang/go/issues/53289#issuecomment-1289743609

It looks to me like these could be different manifestations of the same issue.  Following the advice in the second message, I tried to build Go and run the test suite with `export GO_GCFLAGS="-d=checkptr"` in src/make.bash.  With that, I saw checkptr fail:

fatal error: checkptr: pointer arithmetic result points to invalid allocation

    goroutine 23171 [running]:
    runtime.throw({0x8198ae?, 0xc0001560cf?})
            /home/dap/gotest/gocrash-1667603796953/thread-0-run-0/goroot/src/runtime/panic.go:1047 +0x5d fp=0xc0029f4e70 sp=0xc0029f4e40 pc=0x43c95d
    runtime.checkptrArithmetic(0x770f40?, {0x0, 0x0, 0x0?})
            /home/dap/gotest/gocrash-1667603796953/thread-0-run-0/goroot/src/runtime/checkptr.go:70 +0xea fp=0xc0029f4eb0 sp=0xc0029f4e70 pc=0x40856a
    runtime.lfstackUnpack(...)
            /home/dap/gotest/gocrash-1667603796953/thread-0-run-0/goroot/src/runtime/lfstack_64bit.go:52
    runtime.(*lfstack).pop(...)
            /home/dap/gotest/gocrash-1667603796953/thread-0-run-0/goroot/src/runtime/lfstack.go:47
    runtime.LFStackPop(...)
            /home/dap/gotest/gocrash-1667603796953/thread-0-run-0/goroot/src/runtime/export_test.go:70
    runtime_test.TestLFStack(0xc000d26b60)
            /home/dap/gotest/gocrash-1667603796953/thread-0-run-0/goroot/src/runtime/lfstack_test.go:52 +0x2de fp=0xc0029f4f70 sp=0xc0029f4eb0 pc=0x6deb9e
    testing.tRunner(0xc000d26b60, 0x820dc0)
            /home/dap/gotest/gocrash-1667603796953/thread-0-run-0/goroot/src/testing/testing.go:1446 +0x10b fp=0xc0029f4fc0 sp=0xc0029f4f70 pc=0x4fd94b
    testing.(*T).Run.func1()
            /home/dap/gotest/gocrash-1667603796953/thread-0-run-0/goroot/src/testing/testing.go:1493 +0x2a fp=0xc0029f4fe0 sp=0xc0029f4fc0 pc=0x4fe7ea
    runtime.goexit()
            /home/dap/gotest/gocrash-1667603796953/thread-0-run-0/goroot/src/runtime/asm_amd64.s:1594 +0x1 fp=0xc0029f4fe8 sp=0xc0029f4fe0 pc=0x476121
    created by testing.(*T).Run

That points to this code:

https://github.com/golang/go/blob/39ac1fbd1393deed8888245dcf653220b14376f6/src/runtime/lfstack_64bit.go#L52

So I dug into the checkptr check and added some more printlns and I found that this appears to be a heap address (0xc0001560c0), and “originals” is empty (in checkptrArithmetic).  Given that, I’m not sure how this could ever work, since “originals” seems to come from the compiler (so it doesn’t depend on runtime state).  Can someone point me in the right direction to better understanding what’s wrong here?  Alternatively, is there other information that would be useful to debug this problem?

Thanks,

Dave

[Keith Randall]

I don't think checkptr is intended to be applied blindly inside the runtime. The lfstack code is definitely doing things that checkptr was designed to catch and flag.

Maybe the lfstack routines need a go:nocheckptr annotation?

[Keith Randall]

(Although normally lfstack nodes should not be heap allocated in the first place?)

[Keith Randall]

Should be fixed by https://go-review.googlesource.com/c/go/+/448899

[David Pacheco]

Hi Keith,

Thanks for the helpful (and quick) reply!  I gather you're right about not applying checkptr to Go's own test suite.  I applied your patch, tried again, and started hitting failures in tests that were written for specific past issues.  These seem like false positives.

Do you have other suggestions for how to debug this problem?  I tried checkptr because when I ran the test suite and it failed with a panic message, that's what the panic message from the runtime said to try.

Thanks!

-- Dave

[Keith Randall]

On Thursday, November 10, 2022 at 1:59:11 PM UTC-8 David Pacheco wrote:

Hi Keith,

Thanks for the helpful (and quick) reply!  I gather you're right about not applying checkptr to Go's own test suite.  I applied your patch, tried again, and started hitting failures in tests that were written for specific past issues.  These seem like false positives.

Do you have other suggestions for how to debug this problem?  I tried checkptr because when I ran the test suite and it failed with a panic message, that's what the panic message from the runtime said to try.

That message is intended for people using Go to write their own programs, not for developers of Go itself.

That said, -d=checkptr should work fine for most of all.bash. There's no reason it wouldn't work for say, the compress/gzip tests. But there are certainly tricky cases, like the runtime and the tests of the -d=checkptr feature itself.

Back to the original problem, #53289. The last crash you posted looks like the go tool itself is the one crashing. That tool itself should run fine in checkptr mode. Maybe there's a way to hack the build script to run just that binary in checkptr mode. Maybe just hardcode a "if os.Args[0] == "go" { checkptr = true }" somewhere in runtime startup?