Disallowed conversion uintptr <-> unsafe.Pointer in Go 1.3? or not? confused

[Mateusz Czapliński]

Uh, oh, I didn't notice this earlier, only after reading a related thread here (https://groups.google.com/d/msg/golang-nuts/dIGISmr9hw0/gL4d54xQzhkJ); so then I looked into the release notes at http://tip.golang.org/doc/go1.3#garbage_collector, and found the paragraphs; now I'm confused, could please someone explain to me in simpler words what exactly will be allowed, and what will not? or point me to some already existing thread where this is maybe discussed? I try to follow golang-nuts, but things slip here and there. So, to quote as it stands now:

"Starting with Go 1.3, the runtime assumes that values with pointer type contain pointers and other values do not. This assumption is fundamental to the precise behavior of both stack expansion and garbage collection. Programs that use package unsafe to store uintptrs in pointer values are illegal and will crash if the runtime detects the behavior. Programs that use package unsafe to store pointers in uintptr values are also illegal but more difficult to diagnose during execution. Because the pointers are hidden from the runtime, a stack expansion or garbage collection may reclaim the memory they point at, creating dangling pointers.

Updating: Code that converts a uintptr value stored in memory to unsafe.Pointer is illegal and must be rewritten. Such code can be identified by go vet."

So, are all conversions between "uintptr" and "unsafe.Pointer" disallowed now [that is, in 1.3]? Or is "unsafe.Pointer" -> "uintptr" allowed, but the reverse not? If so, then how can I e.g. safely extract any pointer from WinAPI syscalls? Can't I? Or, er, um, so, um, what is what...? More specifically, I'm personally most concerned now about how, and whether at all, can I try to continue working on https://github.com/akavel/goluago, where I constantly convert something between uintptr and unsafe.Pointer (both ways). And also, again, interested what this means for interacting with WinAPI. Also, besides, curious how this relates to the compatibility promise.

I understand I may be wrong in many places here, and that's specially why I'd be very grateful for any explanations, possibly with examples, also of what's stays OK and legal.

Thanks in advance,
/Mateusz Czapliński.

[Russ Cox]

All code that compiles in 1.2 will compile in 1.3. The question is whether it will run correctly.

In Go 1.2 it was the case that uintptr and unsafe.Pointer were both treated as pointers by the garbage collector. If you did the equivalent of:

    var x uintptr = uintptr(unsafe.Pointer(new(T)))
    runtime.GC()

    p := (*T)(unsafe.Pointer(x))

Then the garbage collection would preserve [not collect] the new(T), because x 'points' at it. uintptr isn't particularly special here, this was true of any pointer-sized integer. 

In Go 1.3 the garbage collector knows the types of specific fields, and it assumes that integer values contain integers (not pointers) and pointer values contain pointers (not integers). So it does not scan x for a pointer and therefore does not keep the new(T) alive, and therefore by the end of the snippet p is pointing to dead memory that may well have been reused for something else. It does not point at a new(T).

There are two conversions above: unsafe.Pointer -> uintptr and uintptr -> unsafe.Pointer. The problem is that in the time between the two conversions, an integer variable (of type uintptr) is being used to store an actual pointer to Go data. Don't do that, and you'll be fine. 

To help detect these situations, 'go vet' flags suspicious 'uintptr -> unsafe.Pointer' conversions. If you 'go get code.google.com/p/go.tools/cmd/vet' you will get the updated checker, even if you are running Go 1.2, and you can run it on your code.

If you are calling a WinAPI syscall that returns a pointer but has return type uintptr, that's not really the right signature for the call. I would suggest fixing the signature if possible. But if the pointer is to Windows-allocated data and not Go-allocated data, then the garbage collector doesn't need to see that pointer, so there is no harm done by having it in a uintptr briefly. That is, a uintptr->pointer conversion is okay if the pointer being created points at memory not managed by Go.

As for compatibility, we've never explicitly specified the guarantees about what you can and can't do with unsafe.Pointer. For the most part we've said 'be reasonable'. The basic idea you need to keep in mind is 'use integers for integers and pointers for pointers' and you'll be fine.

My suggestion is to update your copy of vet, run it on your goluago package, and if there's something flagged that you think should be okay, post the details here in this thread and we can work through them.

Russ

[Mateusz Czapliński]

Thanks a lot for the response.

So from this, am I correct to understand, that as long as I keep my unsafe.Pointer referenced (non-collectable), I can happily convert it to an uintptr and play with the uintptr value as much as I'd like (e.g., pass it to some .c)? I still have some concerns about this usecase, after reading a thread I've just found on golang-dev (https://groups.google.com/d/msg/golang-dev/rd8XgvAmtAA/cI_hXa_FThcJ, which I don't fully understand, unfortunately), related to stacks copying - does/will this feature introduce additional issues for me here?

Related to the pattern above: in goluago I link with Lua C code compiled by Go's cc; the Lua API sometimes takes a "void* userdata", which I used to convert back to some pointer in Go; so, this would still compile in 1.3, but be somehow "illegal"? Under assumption that the premise from my previous section is true, should I rather keep my unsafe.Pointers in some side storage as long as needed (to make uintptrs non-collectable), and also keep a map[uintptr]unsafe.Pointer, so that I could workaround the "illegal" part and, based on uintptr, only retrieve the unsafe.Pointer "legally" from the map? Or, am I OK to work "illegally" here by converting the uintptr received from C back to an unsafe.Pointer (as long as I keep an additional bookmarked reference to the unsafe.Pointer)? Or, none of above applies?

As to WinAPI, yes, I think I mean the case of pointers allocated by Windows. In such case, I feel unsure whether I can convert a uintptr, via unsafe.Pointer, to e.g. *byte, and use this to write a "func peek(base, off uintptr) byte { return *((*byte)(unsafe.Pointer(base+off))) }", or not? At one time, I think I used such a pattern for copying data from a Windows-created buffer byte-by-byte to a []byte. Or, is it "illegal" in some way? Or, will it be illegal with copied stacks? Also, does introducing *byte here introduce a risk of GC trying to collect the data? (Or, crashing with e.g. some "ouch, you showed me a pointer to data that's not mine, and it is not in my maps, naughty, naughty boy"?) Also, I sometimes had, and expect to maybe have, a similar situation in goluago (i.e., of using this "peek()"), when trying to push some results from Lua through the C->Go barrier.

I'll try to run go vet on it, then, but not sure when time will let me, unfortunately. I understand I can use the one from the newest 1.3beta?

Ah, and one more thing: in https://github.com/akavel/rsrc, I create a .syso file with some data, and then create a Slice in C for accessing it, and return it through C->Go barrier as a []byte; is this unsafe, too, because it can make GC try to free that? Now that I think, it may be already unsafe this way even in Go 1.2. (Hm, maybe I should take a []byte argument and fill it? dunno, will have to think more about it probably.)

Thanks,
/Mateusz.

[Ian Lance Taylor]

On Thu, May 22, 2014 at 8:42 AM, Mateusz Czapliński <czap...@gmail.com> wrote:
>
> So from this, am I correct to understand, that as long as I keep my
> unsafe.Pointer referenced (non-collectable), I can happily convert it to an
> uintptr and play with the uintptr value as much as I'd like (e.g., pass it
> to some .c)? I still have some concerns about this usecase, after reading a
> thread I've just found on golang-dev
> (https://groups.google.com/d/msg/golang-dev/rd8XgvAmtAA/cI_hXa_FThcJ, which
> I don't fully understand, unfortunately), related to stacks copying -
> does/will this feature introduce additional issues for me here?

Yes, stack copying can be an issue in unusual cases. If you write
var a [5]int
p = unsafe.Pointer(&a[0])
v := uintptr(p)
f()
p = unsafe.Pointer(v)
then it is possible that the stack will have been copied during the
call to f, which could in some cases mean that p will no longer be
pointing at a.

As long as you retain a pointer in Go, it should be fine to convert a
Go pointer to uintptr and pass it to C. It doesn't matter what C does
with the value. That said, if you have a pointer value, it would in
general be better to pass an unsafe.Pointer to C rather than a
uintptr.

> Related to the pattern above: in goluago I link with Lua C code compiled by
> Go's cc; the Lua API sometimes takes a "void* userdata", which I used to
> convert back to some pointer in Go; so, this would still compile in 1.3, but
> be somehow "illegal"? Under assumption that the premise from my previous
> section is true, should I rather keep my unsafe.Pointers in some side
> storage as long as needed (to make uintptrs non-collectable), and also keep
> a map[uintptr]unsafe.Pointer, so that I could workaround the "illegal" part
> and, based on uintptr, only retrieve the unsafe.Pointer "legally" from the
> map? Or, am I OK to work "illegally" here by converting the uintptr received
> from C back to an unsafe.Pointer (as long as I keep an additional bookmarked
> reference to the unsafe.Pointer)? Or, none of above applies?

I don't think we've resolved the rules as to what should happen if you
have a pointer in Go, pass it to C, and then receive the pointer back
from C. In 1.3 that will work fine assuming the pointer is always
live in Go. However, in the future Go may have a moving garbage
collector. That would break this kind of code if the Go pointer
moves, since the value coming back from C would not have been
adjusted. When and if that happens, we will have to figure out the
rules that will ensure that the pointer passed to C does not move.

Using a map[uintptr]unsafe.Pointer is an interesting idea that might
actually work. But you don't need that for 1.3, although you do of
course need to keep the unsafe.Pointer alive somewhere on the Go side,
and such a map would be one way to do it.

> As to WinAPI, yes, I think I mean the case of pointers allocated by Windows.
> In such case, I feel unsure whether I can convert a uintptr, via
> unsafe.Pointer, to e.g. *byte, and use this to write a "func peek(base, off
> uintptr) byte { return *((*byte)(unsafe.Pointer(base+off))) }", or not?

That's fine. The change in Go 1.3 concerns value stored in memory,
and there is no such value here.

> Ah, and one more thing: in https://github.com/akavel/rsrc, I create a .syso
> file with some data, and then create a Slice in C for accessing it, and
> return it through C->Go barrier as a []byte; is this unsafe, too, because it
> can make GC try to free that? Now that I think, it may be already unsafe
> this way even in Go 1.2. (Hm, maybe I should take a []byte argument and fill
> it? dunno, will have to think more about it probably.)

That's fine. The GC doesn't care about valid pointers that happen to
point into memory allocated by C. It won't try to do anything with
such a pointer.

Ian