Cgo Pointer Arithmetic

[Luke Mauldin]

I have questions about pointer arithmetic illustrated by this Play example:  https://play.golang.org/p/-cZteTY_M2

Questions:

1) Is this the best way to do pointer arithmetic in Go to process a C array and convert it to a Go slice?

2) Go vet gives an error on line 18 but doesn't give much information on how to fix the usage of unsafe.Pointer.  Any recommendations?

[Ian Lance Taylor]

It's usually simplest to write an expression like
s := (*[1 << 20]Tag)(Carr)[:arrLength:arrLength]
That will give you a slice whose backing array is the C array, without
requiring any copying. Of course you then have to make sure that C
array lives on the heap as least as long as the slice does. If that
is an issue, then write
s2 := make([]Tag, arrLength)
copy(s2, s)
To be clear, this assume there are fewer than 1 << 20 entries in the
array, so adjust as needed.


The vet errors for your code are false positives, assuming that cArr
is allocated in C memory. That said, it's easy to avoid them by
writing code like

q := unsafe.Pointer(cArr)
for i := 0; i < arrLength; i++ {
p := (*Tag)(q)
ret = append(ret, int(*p))
q = unsafe.Pointer(uintptr(q) + unsafe.Sizeof(q))
}

The point is: always keep pointers as pointers, except in expressions
that convert to uintptr and back in a single expression.

Ian

[Luke Mauldin]

I modified my example based on the code example you gave at the bottom for the pointer arithmetic and that compiled without any go vet errors.  I attempted to use the slice whose backing array is the C array https://play.golang.org/p/rJoDMu5YAQ but I get a compile error:

main.go:14: cannot convert cArr (type *Tag) to type *[1048576]Tag

Another question, if I update my code to use the slice whose backing array is the C array, I see the slice type supports 1048576 entries. If I only need at most 100 entries, is there a memory penalty imposed by declaring the type to support 1048576 entries?  If so, how great is that penalty?

[Ian Lance Taylor]

On Wed, Aug 31, 2016 at 2:06 PM, Luke Mauldin <lukem...@gmail.com> wrote:
> I modified my example based on the code example you gave at the bottom for
> the pointer arithmetic and that compiled without any go vet errors. I
> attempted to use the slice whose backing array is the C array
> https://play.golang.org/p/rJoDMu5YAQ but I get a compile error:
> main.go:14: cannot convert cArr (type *Tag) to type *[1048576]Tag

Oh, sorry, you need to drop in another conversion to unsafe.Pointer.

> Another question, if I update my code to use the slice whose backing array
> is the C array, I see the slice type supports 1048576 entries. If I only
> need at most 100 entries, is there a memory penalty imposed by declaring the
> type to support 1048576 entries? If so, how great is that penalty?

There is no penalty. It's just a pointer type conversion. Though if
you are seeing that many entries, then I suspect that you left out the
operation that slices the pointer back down to arrLength (the
[:arrLength:arrLength] at the end of line in the code I sent).

Ian

[Luke Mauldin]

After I added the unsafe.Pointer conversion the code compiled: https://play.golang.org/p/QTPyhZzKZH

So my understanding is that line 15 is jut a pointer type conversion to slice that is backed by the C array, so it doesn't matter if I choose 1 << 20 entries or 1 << 22 entries, in both cases the runtime is just creating a slice?

Another question, which of the two approaches (pointer arithmetic vs iterating over sliced back by C array) is more efficient to create a Go slice?  This code will be invoked hundreds of times per minute to convert C arrays coming from C functions into memory safe Go slices that will then be used throughout the Go program.

[James Bardin]

On Thursday, September 1, 2016 at 8:45:02 AM UTC-4, Luke Mauldin wrote:

After I added the unsafe.Pointer conversion the code compiled: https://play.golang.org/p/QTPyhZzKZH

So my understanding is that line 15 is jut a pointer type conversion to slice that is backed by the C array, so it doesn't matter if I choose 1 << 20 entries or 1 << 22 entries, in both cases the runtime is just creating a slice?

Yes, it's just a type conversion, and (*[1 << 20]Tag) is only a type. As long as the size is large enough to hold your data, have the slice operation be within bounds, and be within the size limits of the system, it doesn't matter so much.

 

Another question, which of the two approaches (pointer arithmetic vs iterating over sliced back by C array) is more efficient to create a Go slice?  This code will be invoked hundreds of times per minute to convert C arrays coming from C functions into memory safe Go slices that will then be used throughout the Go program.

Hundreds of times per minute isn't very much. Once you've converted the array to a Go slice, it's accessed the same as any other Go slice. There's no need to calculate the pointer arithmetic by hand when you have the compiler to do that for you.