Change slice type

[greg...@gmail.com]

Hi Guys, quick question...

When i have a buffer

x:=make([]byte, 0, 80000)

Is it possible to use same memory range as []int of size: FLOOR(80000*size_of(byte)/size_of(int)). Basically i want to make slab allocator better

Thanks

[Ian Lance Taylor]

On Tue, Nov 26, 2019 at 5:47 AM <greg...@gmail.com> wrote:
>
> When i have a buffer
> x:=make([]byte, 0, 80000)
>
> Is it possible to use same memory range as []int of size: FLOOR(80000*size_of(byte)/size_of(int)). Basically i want to make slab allocator better

It is possible to do this using the unsafe package. The package docs
(https://golang.org/pkg/unsafe) describe the rules for when this can
be done.

Note in particular that a []byte is not necessarily aligned as a
[]int. You will either have to align the pointers yourself or go the
other way--from []int to []byte.

Ian

[greg...@gmail.com]

Hi Ian, thanks for the help... quick question about alignment because i may be doing something wrong but it shows me that it doesn't matter for the slice type, everything is alignment the same way. Can you comment on this?

On 32 bit it'll always return 4 and for 64 bit - 8... even for char or boolean slices

https://play.golang.org/p/wLbt41sISft

Another question, i create []byte slice, then i create []int slice, in the middle of []byte by changing

    arr := [10]byte{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
    size := len(arr)
    p := uintptr(unsafe.Pointer(&arr))

    var data []int
    sh := (*reflect.SliceHeader)(unsafe.Pointer(&data))
    sh.Data = p + some_alignment_offset_
    sh.Len = size
    sh.Cap = size

Now data points at memory used by arr, when arr will become unreachable... will underlying memory be collected by GC? Should i call some function to inform GC to not collect the memory?

Thanks

[Ian Lance Taylor]

On Wed, Nov 27, 2019 at 9:26 AM <greg...@gmail.com> wrote:
>
> Hi Ian, thanks for the help... quick question about alignment because i may be doing something wrong but it shows me that it doesn't matter for the slice type, everything is alignment the same way. Can you comment on this?

All slices have the same alignment requirement, but slices have an
underlying array, and the alignment requirements of the underlying
array are not the same.

https://play.golang.org/p/s6uFbk-WLkw

> Another question, i create []byte slice, then i create []int slice, in the middle of []byte by changing
>
> arr := [10]byte{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
> size := len(arr)
> p := uintptr(unsafe.Pointer(&arr))

That doesn't look right. That will get you the address of the slice.
You need the address of the underlying array. You need to write
p := uintptr(unsafe.Pointer(&arr[0]))

But that is wrong for another reason. This conversion from
unsafe.Pointer to uintptr is not one of the patterns permitted at
https://golang.org/pkg/unsafe. This approach will not work

> var data []int
> sh := (*reflect.SliceHeader)(unsafe.Pointer(&data))
> sh.Data = p + some_alignment_offset_
> sh.Len = size
> sh.Cap = size
>
> Now data points at memory used by arr, when arr will become unreachable... will underlying memory be collected by GC? Should i call some function to inform GC to not collect the memory?

That is not a problem. Just because arr is unreachable does not mean
that arr's underlying array is unreachable. You could write
p1 := &arr[1]
and then arr could disappear but p1 would still be valid.

Anyhow you need to write this more like (completely untested)

(*[1<<28]int)(unsafe.Pointer((uintptr(unsafe.Pointer(&arr[0])) +
7)&^7)[:size:size]

except that that doesn't handle any alignment issues.

Ian

[Michael Jones]

The general rule -- if there can be a general rule for risk behavior -- is that CPUs like having addresses aligned on an integer multiple of the data element size. So:

access-as-byte data may be on any address (address&(1-1)==0), 

access as 2-byte data on a multiple of two address (address&(2-1)==0),  

access as 4-byte data on a multiple of four address (address&(4-1)==0),  

access as 8-byte data on a multiple of eight address (address&(8-1)==0),  

access as 16-byte data on a multiple of sixteen address (address&(16-1)==0), 

:

(How this might extend to "SIMD vector registers" of various kinds depends on the machine--is it based on the individual component's size, the aggregate size, etc.) There may be an upper limit to this logic of alignment restriction, so maybe 32-bits or 64-bits is always sufficient.

Based on this thinking, the safest way to abuse Go's implicit type safety is to tell Go's allocator that you want something with large data elements (uint64, say) and then use unsafe to view this as 8x as many bytes or 4x as many uint16s.

this what Ian meant.

On Wed, Nov 27, 2019 at 9:26 AM <greg...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/7430d6a7-909e-41fd-a292-5de2be9f9733%40googlegroups.com.

--

Michael T. Jones
michae...@gmail.com

[Jan Mercl]

On Thu, Nov 28, 2019 at 3:16 AM Michael Jones <michae...@gmail.com> wrote:
>
> The general rule -- if there can be a general rule for risk behavior -- is that CPUs like having addresses aligned on an integer multiple of the data element size. So:
>
> access-as-byte data may be on any address (address&(1-1)==0),
> access as 2-byte data on a multiple of two address (address&(2-1)==0),
> access as 4-byte data on a multiple of four address (address&(4-1)==0),
> access as 8-byte data on a multiple of eight address (address&(8-1)==0),
> access as 16-byte data on a multiple of sixteen address (address&(16-1)==0),

It's a bit more complicated for struct fields on 386, some 8 byte data
are 4 byte aligned.