Unsafe Pointer rules

[Carl Mastrangelo]

(to short circuit any question, I have already read https://golang.org/pkg/unsafe/ )

If I create an unsafe.Pointer that points to an invalid memory address, but I never deference it or otherwise pass it along, what happens to it?  Is it a valid go program to just create such a pointer?  The main reason I ask is that I know the GC treats unsafe.Pointer values differently than uintptr.  If the GC were to chase this invalid pointer, it would likely get a segfault.  This means that either the GC knows not to chase such a pointer, or it would chase it and gracefully recover.  

Additionally, if the unsafe.Pointer is pointing to a incorrectly aligned address, the GC could potentially misunderstand and try to walk it.  I'm sure this has been thought of before, but it isn't called out in the docs.

[Ian Lance Taylor]

On Tue, Aug 21, 2018 at 1:19 PM, 'Carl Mastrangelo' via golang-nuts
<golan...@googlegroups.com> wrote:
>
> If I create an unsafe.Pointer that points to an invalid memory address, but
> I never deference it or otherwise pass it along, what happens to it?

If you never deference it and never do anything with it, then in

practice it most likely gets eliminated by the compiler. That said:

> Is it a valid go program to just create such a pointer?

No. The only way you could create such a Pointer is by converting
from uintptr, or, essentially equivalently, by calling C or assembler
code. The unsafe package docs explain all the cases in which it is
permitted to convert a uintptr to an unsafe.Pointer. Using any other
mechanism is invalid. The permitted mechanisms never produce an
unsafe.Pointer that contains an invalid memory address.

> The main reason I ask is
> that I know the GC treats unsafe.Pointer values differently than uintptr.
> If the GC were to chase this invalid pointer, it would likely get a
> segfault. This means that either the GC knows not to chase such a pointer,
> or it would chase it and gracefully recover.

In practice, in the current implementation, what will happen is that
the GC will attempt to find the object to which the pointer points,
will fail, and will crash with a "pointer to unallocated span" error.

> Additionally, if the unsafe.Pointer is pointing to a incorrectly aligned
> address, the GC could potentially misunderstand and try to walk it. I'm
> sure this has been thought of before, but it isn't called out in the docs.

This isn't an issue with the current implementation. The current GC
doesn't care about the type of the pointer, so pointers have no
alignment requirements as far as the GC is concerned. Every pointer
is effectively a *byte for GC purposes.

Ian

[Carl Mastrangelo]

The answer must be more nuanced than that, because it is possible to take a nil pointer and construct an unsafe.Pointer from it.  

The reason I am interested in this is (and please don't judge too early) is I'm toying around with implementing some atomic primitives.  In particular, I would like to play around with with the cmpxchg16b instruction which needs 16 byte alignment.   Go does not provide a way to enforce a data structure has such alignment, so I am attempting to define a struct that I can index into.  (assume 64bit words).  For example, the datastructure I want is this:

// alignment of foo is 16

type foo struct {

  uintptr

  unsafe.Pointer

}

But I can't assert this.  The next best thing is to make a struct 2x the size, and make a pointer to the first aligned part:

type foo struct {

  [4]uintptr

}

This way I can get an aligned address pointing into the middle of this array for using cmpxchg16b.   The problem with this is that if any of the interior values are not seen a pointers by the GC.  In order to keep the values alive they need to be unsafe.Pointer:

type foo struct {

  [4]unsafe.Pointer

}

Now this is a problem.  There is really only one pointer in here, the other value is just some arbitrary bytes.  Since the GC now things the addresses are real, it will crash.  What is the correct way to get an aligned struct that contains pointers?

[Ian Lance Taylor]

On Tue, Aug 21, 2018 at 4:08 PM, 'Carl Mastrangelo' via golang-nuts
<golan...@googlegroups.com> wrote:
>
> The answer must be more nuanced than that, because it is possible to take a
> nil pointer and construct an unsafe.Pointer from it.

Yes, OK, nil is an exception, as it is for any pointer type.

> The reason I am interested in this is (and please don't judge too early) is
> I'm toying around with implementing some atomic primitives. In particular,
> I would like to play around with with the cmpxchg16b instruction which needs
> 16 byte alignment. Go does not provide a way to enforce a data structure
> has such alignment, so I am attempting to define a struct that I can index
> into. (assume 64bit words). For example, the datastructure I want is this:
>
> // alignment of foo is 16
> type foo struct {
> uintptr
> unsafe.Pointer
> }
>
> But I can't assert this. The next best thing is to make a struct 2x the
> size, and make a pointer to the first aligned part:
>
> type foo struct {
> [4]uintptr
> }
>
> This way I can get an aligned address pointing into the middle of this array
> for using cmpxchg16b. The problem with this is that if any of the interior
> values are not seen a pointers by the GC. In order to keep the values alive
> they need to be unsafe.Pointer:
>
> type foo struct {
> [4]unsafe.Pointer
> }
>
> Now this is a problem. There is really only one pointer in here, the other
> value is just some arbitrary bytes. Since the GC now things the addresses
> are real, it will crash. What is the correct way to get an aligned struct
> that contains pointers?

There isn't one. There is some discussion at https://golang.org/issue/19057 .

Ian

[peterGo]

On Tuesday, August 21, 2018 at 7:08:43 PM UTC-4, Carl Mastrangelo wrote:

The answer must be more nuanced than that, because it is possible to take a nil pointer and construct an unsafe.Pointer from it.  

The reason I am interested in this is (and please don't judge too early) is I'm toying around with implementing some atomic primitives.  In particular, I would like to play around with with the cmpxchg16b instruction which needs 16 byte alignment.   Go does not provide a way to enforce a data structure has such alignment, so I am attempting to define a struct that I can index into.  (assume 64bit words).  For example, the datastructure I want is this:

// alignment of foo is 16

type foo struct {

  uintptr

  unsafe.Pointer

}

 Carl,

Here is a simple attempt to provide you with a pointer to a 16-byte aligned Foo struct for your cmpxchg16b instruction experiments. See the NewFoo() function. Fields, including hidden fields, are initialized to the zero value for the type: zero for uintptr and nil for unsafe.Pointer.

Playground: https://play.golang.org/p/_nVltTdS5kS

Peter

[Louki Sumirniy]

It seems to me the only way to achieve this would be to allocate a []byte twice the size you need, to be sure, then get the address of the start and offset it (if necessary) until it is a number (as in uintptr) divisible by 16, then it would be correctly structured. I would think you will find that you probably have to step quite outside of the normal go runtime to achieve this. Probably it would be better to write the primitives in C and then make glue to connect to it and stuff to make sure it's freed up at exit.

[keith....@gmail.com]

Just to add, it is ok to have a pointer (unsafe.Pointer or otherwise) to unreadable memory.  For instance, you can use syscall.Mmap and syscall.Mprotect to produce such a state.  The GC will not dereference any pointer that points outside the Go heap.

Public Service Announcement #1: Don't use syscall.Mprotect on the Go heap.

Public Service Announcement #2: It has to be mapped.  It is not ok to have a pointer to unmapped memory, as Go might later allocate that address for the Go heap.