unexpected fault address 0xfffffff800000019

[Andrew Werner]

A situation has arisen a few times lately where we've seen segfaults that have strange looking address values. Another thing that makes the situation confusing is that the memory referenced by the stack trace should have been allocated at init time. This seems to only be happening on macOS. The code in question was built using go1.15.5 on x86_64-apple-darwin20.3.0. We don't have a reliable repro but it has happened more than once. A user did report seeing an odd message involving `g->` or so they claim. From my sleuthing, the only code which could throw that message would be: https://github.com/golang/go/blob/go1.15.5/src/runtime/proc.go#L703. 

Anyway, this all smells of memory corruption to me but I would love somebody to confirm my feeling that that fault address feels bogus. Is there any non-corruption reason why one might see this sort of fault?

unexpected fault address 0xfffffff800000019
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0xfffffff800000019 pc=0x5658770]

goroutine 2093790 [running]:
runtime.throw(0x865aaf3, 0x5)
	/usr/local/opt/go/libexec/src/runtime/panic.go:1116 +0x72 fp=0xc0338bdd18 sp=0xc0338bdce8 pc=0x403c7b2
runtime.sigpanic()
	/usr/local/opt/go/libexec/src/runtime/signal_unix.go:749 +0x3e5 fp=0xc0338bdd48 sp=0xc0338bdd18 pc=0x40532e5

Cockroach is tracking this in https://github.com/cockroachdb/cockroach/issues/62283. 

[awer...@gmail.com]

Oh, we've got another confirmed case of the scary runtime error. I guess now the question is, does anybody know of any memory corruption bugs in macOS and go1.15.5? I didn't see anything obvious.


runtime: gp: gp=0xc000802900, goid=0, gp->atomicstatus=0
runtime: g: g=0xc000602480, goid=0, g->atomicstatus=0
fatal error: bad g->status in ready

https://gist.github.com/awoods187/e9289c706ddc529034368fd3a2a25323

[Kurtis Rader]

On Wed, Mar 24, 2021 at 7:05 PM awer...@gmail.com <awer...@gmail.com> wrote:

Oh, we've got another confirmed case of the scary runtime error. I guess now the question is, does anybody know of any memory corruption bugs in macOS and go1.15.5? I didn't see anything obvious.

I'm unaware of any memory corruption bugs inimical to Go on macOS. Note that you can use the vmmap(1) command to examine the memory map of a process on macOS. On my system running that against an Elvish process (a shell written in Go) does not show any addresses with a 0xffffff prefix. So the 0xfffffff800000019 address is suspicious on its face. The first half would be -8 interpreted as an int32 and the second half would be 25 (decimal). You say the "memory referenced by the stack trace should have been allocated at init time". What do you mean by that? Are you saying you explicitly mmap memory at 0xfffffff80000000 (or a base address in the general range)? That doesn't seem likely so I don't know how to interpret your assertion.

Looking at your project it appears you are using CGO which means the most likely cause of the corruption is the linked C code.

--

Kurtis Rader

Caretaker of the exceptional canines Junior and Hank

[Kurtis Rader]

On Wed, Mar 24, 2021 at 8:25 PM Kurtis Rader <kra...@skepticism.us> wrote:

On Wed, Mar 24, 2021 at 7:05 PM awer...@gmail.com <awer...@gmail.com> wrote:

Oh, we've got another confirmed case of the scary runtime error. I guess now the question is, does anybody know of any memory corruption bugs in macOS and go1.15.5? I didn't see anything obvious.

I'm unaware of any memory corruption bugs inimical to Go on macOS. Note that you can use the vmmap(1) command to examine the memory map of a process on macOS. On my system running that against an Elvish process (a shell written in Go) does not show any addresses with a 0xffffff prefix. So the 0xfffffff800000019 address is suspicious on its face. The first half would be -8 interpreted as an int32 and the second half would be 25 (decimal). You say the "memory referenced by the stack trace should have been allocated at init time". What do you mean by that? Are you saying you explicitly mmap memory at 0xfffffff80000000 (or a base address in the general range)? That doesn't seem likely so I don't know how to interpret your assertion.

Also, values like 8 (and -8, 4, -4) are particularly suspect in problems of this nature since they suggest invalid pointer arithmetic.

[Amnon]

I would check if it reproduces on Go 1.16.2

[Kurtis Rader]

On Thu, Mar 25, 2021 at 7:34 AM Amnon <amn...@gmail.com> wrote:

I would check if it reproduces on Go 1.16.2

Testing whether a problem exists with the most release of a particular command (or project/ecosystem) is generally good advice. However, I don't see how your reply is useful, @Amnon Baron Cohen, in this context. Had you pointed to a particular issue resolved between the 1.15.15 release, just four months ago, I would have agreed with your suggestion to attempt to reproduce the problem with the most recent Go release.