efence and changing the read-validity of memory pages on darwin/amd64

[Jason E. Aten]

go1.14.4 / darwin / amd64

I'm tracking down an issue where a certain code (not sure where, in large legacy code) 

is touching memory that it should not.

I thought of the old electric fence technique from C. The electric fence memory allocator would put each object

on its own page on malloc(). Then on free(), it would mark the page as invalid, so that the next time

user code tried to read from the memory that had already been freed, an immediate segfault

would give us a stack trace and tell those of us debugging exactly where the bad access had

come from.

I'd like to do the same in Go.

GODEBUG=efence=1

says, quoting from https://golang.org/pkg/runtime/

efence: setting efence=1 causes the allocator to run in a mode
where each object is allocated on a unique page and addresses are
never recycled.

so the first part is already available in Go. Now I just need your assistance figuring out the 2nd part.

How can go code take the pointer address of an object, and mark that page as "invalid" so that

any read of it will segfault?

Thanks!

[Ian Lance Taylor]

In efence mode that already happens for large allocations. See the
calls to sysFault in the runtime package. I guess the additional step
would be to treat every allocation as a large allocation. You would
burn through memory quite quickly, though.

Ian

[Evan Jones]

I just spent a solid day debugging memory corruption with a Cgo library that was passing a pointer as uintptr, which failed in rare cases when the stack was copied then overwritten. The GODEBUG=efence=1 flag actually made the bug go away ... unless you make the Goroutine stacks big enough to trigger this sort of stack protection.

Two changes to efence that might be helpful for these types of bugs:

1. Make efence overwrite small allocations that are freed, to try and make memory corruption bugs more obvious.

2. Make efence increase the default Goroutine stack size to 1 page, so it can always mark stacks as inaccessible

3. Randomize the stack starting point, to trigger stack growth in different places in different runs of the program.

If these seem like reasonable suggestions, I would be happy to write a Go issue tracker item with them, and more details about my experience using this.

Evan Jones

[Keith Randall]

On Tuesday, February 9, 2021 at 11:05:23 AM UTC-8 Evan Jones wrote:

I just spent a solid day debugging memory corruption with a Cgo library that was passing a pointer as uintptr, which failed in rare cases when the stack was copied then overwritten. The GODEBUG=efence=1 flag actually made the bug go away ... unless you make the Goroutine stacks big enough to trigger this sort of stack protection.

Two changes to efence that might be helpful for these types of bugs:

1. Make efence overwrite small allocations that are freed, to try and make memory corruption bugs more obvious.

This is what GODEBUG=clobberfree=1 already does. Maybe efence mode can enable that as well? Or you can just use both.

 

2. Make efence increase the default Goroutine stack size to 1 page, so it can always mark stacks as inaccessible

Sure. Just to be clear, you'd want to override the minimum stack size, not the starting stack size, as stacks can shrink.

 

3. Randomize the stack starting point, to trigger stack growth in different places in different runs of the program.

Not sure about this one. It would find errors, but then it would make them very hard to reproduce.