Computing a hash over a uintptr ?

[christoph...@gmail.com]

The documentation specifies that a pointer can be cast into a uintptr integer. I assume that the uintptr is ignored by the garbage collector. 

I would like to compute a hash over the uintptr. How do I achieve that ?

The documentation doesn’t specify that we can cast an uintptr into a uint64. The byte size may not match. What would be the idiomatic way to compute a hash over a uintpr ? Is it valid to do that ?

To give some context, the application is for a cache with the hash key computed over a pointer to a heap allocated struct. I need to store the pointer in the cache entry to test for matching, but this would prevent the struct to be garbage collected. Storing the pointer as an uintptr would do the trick.

[Axel Wagner]

On Thu, Jun 3, 2021 at 4:19 PM christoph...@gmail.com <christoph...@gmail.com> wrote:

The documentation doesn’t specify that we can cast an uintptr into a uint64.

Both are integer types, so they can be converted. You might be worried about their respective sizes and you are correct that the spec does not guarantee that a uintptr is at most 64 bit. However, at least for any implementation I know on any supported architecture, that's the case. If you *really* want to make sure, you can do something like this:

func toBytes(v uintptr) []byte {

    var out []byte
    mask := ^uintptr(0)
    for mask != 0 {
        out = append(out, uint8(v))
        v >>= 8
        mask >>= 8
    }
    return out
}

 

But really, you can just trust that a uintptr fits into a uint64. Or, if you want to future-proof, assume it's uint64, but guard by build tags to known architectures (so it at least fails to compile if that assumption ever changes).

To give some context, the application is for a cache with the hash key computed over a pointer to a heap allocated struct. I need to store the pointer in the cache entry to test for matching, but this would prevent the struct to be garbage collected. Storing the pointer as an uintptr would do the trick.

This is problematic. The address of a value can change, so the uintptr would change as well. So there is no guarantee that your hash stays the same, in subsequent calls.

Today, that usually doesn't happen, but there is no guarantee it stays that way. If you are willing to assume that it doesn't happen, you should definitely also be willing to assume a uintptr fits into a uint64

 

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/7983a13f-5bf6-4299-a598-1d023ec9a9e9n%40googlegroups.com.

[Robert Engels]

Doesn’t that depend on what the uintptr refers to? Eg if it was allocated off heap to begin with then it should be stable  and computing a hash on it is valid. 

On Jun 3, 2021, at 9:42 AM, 'Axel Wagner' via golang-nuts <golan...@googlegroups.com> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAEkBMfFsZNHLXcONF9C6Nadr6muUP_kgOJwM3QUXSZ0KxPnfYQ%40mail.gmail.com.

[Ian Lance Taylor]

On Thu, Jun 3, 2021 at 9:13 PM Robert Engels <ren...@ix.netcom.com> wrote:
>
> Doesn’t that depend on what the uintptr refers to? Eg if it was allocated off heap to begin with then it should be stable and computing a hash on it is valid.

That is true in current implementations, but Go, the language, does
not guarantee that pointers will never move.

Ian

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/35E8DBC1-E067-4717-BFFF-18732D77B987%40ix.netcom.com.

[Robert Engels]

I was referring to the “special cases” in this section https://golang.org/cmd/cgo/ - wouldn’t all of these uintptr be stable?

On Jun 4, 2021, at 12:25 AM, Ian Lance Taylor <ia...@golang.org> wrote:

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAOyqgcWY%3DS%3DtriGaYtNK_4v4Nw5AAZBCFOY2hKyTHQ9guaaiTw%40mail.gmail.com.

[christoph...@gmail.com]

> That is true in current implementations, but Go, the language, does not guarantee that pointers will never move.

That is what I thought, but it is allowed to use a pointer far map keys. And I have seen some programs/package using this feature.

Apparently all these programs and packages would break if objects were to be moved around ?

[Axel Wagner]

No, because the Garbage Collector knows that they are pointers and would change the pointer in the map (and presumably its bucket) if it moved the value. It's certainly not trivial to implement, but these are problems that would have to be solved if the GC starts moving things around - because the language allows to use pointers as keys, so we need to keep that safe.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/eea09044-6db3-450b-ba6e-0c31dd733681n%40googlegroups.com.

[Jan Mercl]

On Fri, Jun 4, 2021 at 3:02 PM christoph...@gmail.com
<christoph...@gmail.com> wrote:

> > That is true in current implementations, but Go, the language, does not guarantee that pointers will never move.
>
> That is what I thought, but it is allowed to use a pointer far map keys. And I have seen some programs/package using this feature.
> Apparently all these programs and packages would break if objects were to be moved around ?

u-int-ptr is just an Unsigned Integer the size of a Pointer. Nothing
conceptually different wrt to int, uint, int64, ...

It has no special properties wrt to pointers. Unsafe conversion
from/to uintptr and unsafe.Pointers is supported, yes, but as soon as
the value of the pointer becomes an uintptr, the connection is lost.
Sometimes the disconnection does not matter because the pointer is
known to be stable. Like the address of C-malloced memory, or C-mmaped
memory etc. However, Go pointers to Go objects allocated by the Go
runtime can move and _do_ move in ways that are not specified. So
correct code cannot rely on such stability.

[Ian Lance Taylor]

On Fri, Jun 4, 2021 at 5:23 AM Robert Engels <ren...@ix.netcom.com> wrote:
>
> I was referring to the “special cases” in this section https://golang.org/cmd/cgo/ - wouldn’t all of these uintptr be stable?

The cgo tool gets special treatment: each cgo call is permitted to pin
a known number of pointers. There is some background at
https://go.googlesource.com/proposal/+/refs/heads/master/design/12416-cgo-pointers.md.

There is a similar exception for calls to syscall.Syscall, documented
at https://golang.org/pkg/unsafe/#Pointer.

These mechanisms are specific to those uses, and don't apply to all
pointers in general.

Ian

[Robert Engels]

I was thinking of using C.malloc to create off heap structures. The uintptr to those structs should be stable and can be used as map keys or values, right?

Not suggesting anyone should do that - just feasibility.

> On Jun 4, 2021, at 10:38 AM, Ian Lance Taylor <ia...@golang.org> wrote:

[Ian Lance Taylor]

On Fri, Jun 4, 2021 at 10:00 AM Robert Engels <ren...@ix.netcom.com> wrote:
>
> I was thinking of using C.malloc to create off heap structures. The uintptr to those structs should be stable and can be used as map keys or values, right?
>
> Not suggesting anyone should do that - just feasibility.

Yes, that works reliably. It's only Go pointers (pointers to memory
allocated by Go) that can potentially change.