Is there a way that can share an array int []byte type with a []int type variable? Or how to cast a pointer type to an array type?

[Andrew Lee]

Hey, everyone:

i have a requirement: i create a buffer int []byte type, and i want to store some meta info into the last half of the buffer. The meta info is integer typed. So i want to do this if in C language:

byte[] buffer = (*byte)malloc(1024)
int[] meta = (int *)(void *)(buffer + 512)


meta[0] = 0
meta[1] = 1
...

But how to do it in Golang:

import "unsafe"
var buffer []byte =  new([]byte, 1024)
???

The only way that i find possible is:

var firstMetaAdress uintptr = uinptr(unsafe.Pointer(&buffer [512]))

*( (*int32) (unsafe.Pointer(firstMetaAdress) ) )= 0
*( (*int32) (unsafe.Pointer(firstMetaAdress + 4) ) )= 1
...

Except the pointer arithmetic, the two embedded casst are necessary. So i have no idea how many overheads these casts will cost. Is it the same as the C language that i can ignore? 

All thanks for your help and questions.

[chris dollin]

How about

type Buffer struct {
bytes [512]byte
ints [128]int32
}

And now if b is a Buffer b.bytes[:] is a slice of the bytes
and b.ints[:] is a slice of the ints.

No unsafe needed.

(Maybe you really want to do something else but I'd not
attempt such a superficial translation from C.)

Chris

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Chris "allusive" Dollin

[Ian Lance Taylor]

On Thu, Sep 17, 2015 at 3:07 AM, Andrew Lee <lichu...@gmail.com> wrote:
>
> The only way that i find possible is:
> var firstMetaAdress uintptr = uinptr(unsafe.Pointer(&buffer [512]))
> *( (*int32) (unsafe.Pointer(firstMetaAdress) ) )= 0
> *( (*int32) (unsafe.Pointer(firstMetaAdress + 4) ) )= 1

That is completely unsafe. Since Go is a GC language, you must not
retain real pointer values as non-pointer types. That will break.
You should get a warning from "go vet" about this.

What you could do is this:

p := (*[64]int)(unsafe.Pointer(&buffer[512]))

Then p is a pointer to an int array that you can use to index into the
second half of the buffer.

Needless to say, this is unsafe. That's why you can only do it if you
import the unsafe package. It would be a lot better to write this in
a type safe way.

Ian

[Andrew Lee]

thanks, it's feasible that i know the fix size of the buffer. But the buffer's size depends on a parameter, either is the size of the second half of the buffer. 

So if i want to do this, maybe i should declare a fix-size that the buffer's size can't exceed. Such as:

import "unsafe"
const maxSize = 2048
var size = 1024
var buffer []byte =  new([]byte, size)

p := (*[maxSize/4/2]int32)(unsafe.Pointer(&buffer[len(buffer/2)]))

p[0] =
...
p[size/4/2] =

Is it safe to do this if i watch out.

The second question is:

If i follow the safe type way in Golang, i have no idea to share a memory with two different types. Of course i can declare a new memory to store the half info, and then copy it to the byte array finally. But it seems not less efficient than share it commonly in C language. Am i right?

在 2015年9月18日星期五 UTC+8上午6:17:55，Ian Lance Taylor写道：

[Andrew Lee]

I attempt to share a memory between the byte array and int array. Your code seems to miss the target.

在 2015年9月17日星期四 UTC+8下午10:15:01，ehedgehog写道：

[Jan Mercl]

On Fri, Sep 18, 2015 at 11:00 AM Andrew Lee <lichu...@gmail.com> wrote:

> If i follow the safe type way in Golang, i have no idea to share a memory with two

> different types.

Because everything is represented as a finite number of memory bytes and []byte is just that, everything can be put to []byte and restored later. For example, an int32 is four memory bytes. It's enough to write/read those 4 bytes into/from the []byte. You can do it manually as 4 assignments or use encoding/binary for that.

PS:

Replace

        var buffer []byte =  new([]byte, size)

with

        var buffer = make([]byte, size)

-j

--

-j

[dja...@gmail.com]

something like this ?

http://play.golang.org/p/8FM7yRFTyp

Djadala

[chris dollin]

On 18 September 2015 at 10:03, Andrew Lee <lichu...@gmail.com> wrote:
> I attempt to share a memory between the byte array and int array. Your code
> seems to miss the target.

It's not clear to me why sharing the memory is a /requirement/
rather than a tactic you've used to achieve some end.

What are you actually trying to /do/ with this buffer? What's the
benefit of the sharing (other than making it look like C)?

Chris

--
Chris "allusive" Dollin

[Andrew Lee]

Yep, it is feasible. But there are overheads transforming between 4 bytes and 1 int. If bypassing the type system is permitted, the compiler system can use the low-level instructions to transfer 4 bytes to 1 integer.

C language can do this by transfer *byte to *int through intermediate type *void. Then getting a integer just need a instrcution.

在 2015年9月18日星期五 UTC+8下午5:37:25，dja...@gmail.com写道：

[Giulio Iotti]

On Friday, September 18, 2015 at 11:37:25 AM UTC+2, dja...@gmail.com wrote:

something like this ?

http://play.golang.org/p/8FM7yRFTyp

The standard library can help: https://play.golang.org/p/02XBMi6yx1

This can be useful when writing raw binary data (for sharing with a C program, for example.)

-- 

Giulio Iotti

[Giulio Iotti]

On Friday, September 18, 2015 at 12:45:39 PM UTC+2, Andrew Lee wrote:

Yep, it is feasible. But there are overheads transforming between 4 bytes and 1 int. If bypassing the type system is permitted, the compiler system can use the low-level instructions to transfer 4 bytes to 1 integer.

Is int 4 bytes? Be careful with types.

As they have already asked you, why do you need this? Are you writing the slices as bytes to a file/socket?

-- 

Giulio Iotti  

[Andrew Lee]

Good question. I use this byte buffer to storage Key-Value(variable size) paris, and then transfer it to other same processes. 

In order to deserialize it in receiving side, i need to store some meta info into the buffer in sending side.

In previous version, i allocate a isolated int array to store the meta info. I dosen't serialize the meta into somewhere in the byte array until i will send it.

Then it occurred to me that i can use C style way to share a same sequential space. If i can use a int array that point somewhere in the byte array, i can assign meta info directly to the indicated address in the byte array.

Many guys advise me to use encoding/binary to serialize and deserialize the bytes. But it need computing overheads.

Andrew Lee

在 2015年9月18日星期五 UTC+8下午6:01:24，ehedgehog写道：

[Andrew Lee]

It is feasible. As i said above, it need some computing overheads to transform between bytes and integers.

If i want pursue exceeding optimization, i think the issue is worth talking.

在 2015年9月18日星期五 UTC+8下午7:00:25，Giulio Iotti写道：

[Giulio Iotti]

On Friday, September 18, 2015 at 1:11:51 PM UTC+2, Andrew Lee wrote:

It is feasible. As i said above, it need some computing overheads to transform between bytes and integers.

If i want pursue exceeding optimization, i think the issue is worth talking.

Absolutely yes. If you want to write this data, implement a writer on top of Djadala's example. That one didn't have any overhead (allocation). Mine does.

-- 

Giulio Iotti 

[⚛]

On Friday, September 18, 2015 at 12:45:39 PM UTC+2, Andrew Lee wrote:

Yep, it is feasible. But there are overheads transforming between 4 bytes and 1 int. If bypassing the type system is permitted, the compiler system can use the low-level instructions to transfer 4 bytes to 1 integer.

C language can do this by transfer *byte to *int through intermediate type *void. Then getting a integer just need a instrcution.

Based on my experiments, it is doable for a compiler to recognize (uint32(s.b[ii]) | uint32(s.b[ii+1])<<8 | uint32(s.b[ii+2])<<16 | uint32(s.b[ii+3])<<24) and emit a single x86 instruction to do the load (plus a boundary check).

Unfortunately, it is my impression that Google Go team has no plans to develop such a compiler. This is my personal opinion, you don't have to agree with it, and I may be wrong about it.

The current Go compiler can for example transform 2 shifts into a single rotation instruction, but when I checked this some year ago I realized that it is just hard-coded and it cannot be said that the Go compiler supports "transforming 2 shifts into a single rotation" in a generic way. I don't know whether this is still true for version 1.5.

[Ian Lance Taylor]

On Fri, Sep 18, 2015 at 4:08 AM, Andrew Lee <lichu...@gmail.com> wrote:
>
> Good question. I use this byte buffer to storage Key-Value(variable size)
> paris, and then transfer it to other same processes.

How are you transferring the buffer?

Ian