[CGO] how to pass a 2d slice to C?

[Cobb Liu]

My code is :

package main

/*                                                                                                                                                                                                                

#include <stdio.h>                                                                                                                                                                                                

#include <string.h>                                                                                                                                                                                               

                                                                                                                                                                                                                  

void fill_2d_array(char (*s)[16]) {                                                                                                                                                                               

    strcpy(s[0], "hello");                                                                                                                                                                                        

    strcpy(s[1],"cgo");                                                                                                                                                                                           

}                                                                                                                                                                                                                 

*/

import "C"

import "fmt"

import "unsafe"

func main() {

        dirs := make([][]byte, 4)

        for i := 0; i < 4; i++ {

                dirs[i] = make([]byte, 16)

        }

        C.fill_2d_array(((*C.char)[16])(unsafe.Pointer(&dirs)))

        fmt.Println(dirs)

}

when I run with 'go run test.go', it failed and said:

# go run test.go

./test.go:21: type *C.char is not an expression 

How can I pass a 2 dimensional slice to a C function like 'fill_2d_array' above as a parameter? 

Thanks.

[Ian Lance Taylor]

This code looks wrong at several levels.

It looks like you are trying to write a Go type (*C.char)[16]. That's
not a Go type. A Go type would be [16]*C.char. But you need a
pointer, so you probably want *[16]*C.char. But that doesn't match
your C code. Your C code is expecting a pointer to an array of 16
chars, which in Go would be written as *[16]C.char. The strcpy to
s[0] is working because in C arrays decay to pointers, so the C code
is effectively **C.char and s[0] is *C.char. But none of this matches
what Go is passing, since a slice is not a pointer.

Ian

[Cobb Liu]

Thanks for your reply.

Then how can I pass a Cgo's 2d array to C function, and the C function fills the array? Is there is a best practice to do it?

[Cobb Liu]

I did a trick in my codes below and it works:

package main

/*                                                                                                                                                                                                                
#include <stdio.h>                                                                                                                                                                                                
#include <string.h>                                                                                                                                                                                              
                                                                                                                                                                                                                 

void fill_array(char *s) {                                                                                                                                                                                        
    strcpy(s, "abc");                                                                                                                                                                                            
}                                                                                                                                                                                                                
void fill_2d_array(char **arr, int columeSize) {                                                                                                                                                                  
    strcpy((char*)(arr+0*sizeof(char)*columeSize), "hello");                                                                                                                                                      
    strcpy((char*)(arr+1*sizeof(char)*columeSize/sizeof(char*)), "go");                                                                                                                                          
}                                                                                                                                                                                                                
*/

import "C"
import "fmt"
import "unsafe"


func main() {

        var dir [10]byte
        C.fill_array((*C.char)(unsafe.Pointer(&dir[0])))
        fmt.Println(string(dir[:]))

                                                                                                                                                                                 
        dirs := make([][]byte, 4)
        for i := 0; i < 4; i++ {
                dirs[i] = make([]byte, 16)
        }

        C.fill_2d_array((**C.char)(unsafe.Pointer(&dirs[0][0])), C.int(16))
        fmt.Println(dirs)              
}

Anyone has  better ways?

[jianzh...@gmail.com]

Hey Guys,

I'm in trouble in the same issue. My code as the following:

test.go

```go

    name := []string{"gpu0", "gpu1", "gpu2", "gpu3"}

    matrix := [3][3]int{{1, 0, 0}, {3, 3, 0}, {3, 3, 2}}

    C.test_settopologyresource(mod, C.CString("node1"), C.int(2), (**C.char)(unsafe.Pointer(&name[0])), (**C.int)(unsafe.Pointer(&matrix[0][0])))

```

xxx.c

```c

bool test_settopologyresource(bsagomodule* _self, char* _nodename,

        int _resourceindex, char** _nameunits, int** _levelmatrix) {

    printf("\n set topology resource-> node:%s, resource index:%d", _nodename, _resourceindex);

    bool result = settopologyresource(_self->client, _nodename, _resourceindex, _nameunits, _levelmatrix);

    return result;

}

```

But when I execute `go run test.go`, got the below errors:

```

go run test.go

fatal error: unexpected signal during runtime execution

[signal SIGSEGV: segmentation violation code=0x1 addr=0xc pc=0x7f5e655089da]

runtime stack:

runtime.throw(0x4b1190, 0x2a)

/usr/local/go/src/runtime/panic.go:596 +0x95

runtime.sigpanic()

/usr/local/go/src/runtime/signal_unix.go:274 +0x2db

goroutine 1 [syscall, locked to thread]:

runtime.cgocall(0x481400, 0xc42004f668, 0x4b0f76)

/usr/local/go/src/runtime/cgocall.go:131 +0xe2 fp=0xc42004f628 sp=0xc42004f5e8

main._Cfunc_bsagomodule_settopologyresource(0x1226440, 0x122d2f0, 0x2, 0xc42001c280, 0xc42005c050, 0x122d200)

command-line-arguments/_obj/_cgo_gotypes.go:174 +0x4a fp=0xc42004f668 sp=0xc42004f628

main.main.func8(0x1226440, 0x122d2f0, 0x2, 0xc42001c280, 0xc42005c050, 0x0)

/home/zhangjian/project/k8s_bsamodule/test/test.go:94 +0xf4 fp=0xc42004f6a8 sp=0xc42004f668

main.main()

/home/zhangjian/project/k8s_bsamodule/test/test.go:96 +0xeca fp=0xc42004ff88 sp=0xc42004f6a8

runtime.main()

/usr/local/go/src/runtime/proc.go:185 +0x20a fp=0xc42004ffe0 sp=0xc42004ff88

runtime.goexit()

/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1 fp=0xc42004ffe8 sp=0xc42004ffe0

goroutine 17 [syscall, locked to thread]:

runtime.goexit()

/usr/local/go/src/runtime/asm_amd64.s:2197 +0x1

exit status 2

```
Have any ways to solve this problem?

Thanks a lot!

在 2015年12月10日星期四 UTC+8上午3:07:17，Ian Lance Taylor写道：

[jianzh...@gmail.com]

I have changed the go code into the below, but still got error. Something else I missed?

    gonameunits := []string{"gpu0", "gpu1", "gpu2", "gpu3"}

    nameunits := make([]*C.char, len(gonameunits))

    for i, _ := range gonameunits {

        nameunits[i] = C.CString(gonameunits[i])

        defer C.free(unsafe.Pointer(nameunits[i]))

    }

    fmt.Println("nameunits:", nameunits)

    golevelmatrix := [3][3]int{{1, 0, 0}, {3, 3, 0}, {3, 3, 2}}

    levelmatrix := make([][]C.int, len(golevelmatrix))

    for i, _ := range golevelmatrix {

        levelmatrix[i] = make([]C.int, len(golevelmatrix[i]))

        for j, _ := range golevelmatrix[i] {

            levelmatrix[i][j] = C.int(golevelmatrix[i][j])

        }

    }

    fmt.Println("levelmatrix:", levelmatrix)

    C.test_settopologyresource(mod, C.CString("node1"), C.int(2), (**C.char)(unsafe.Pointer(&nameunits[0])), (**C.int)(unsafe.Pointer(&levelmatrix[0][0])))

[Konstantin Khomoutov]

On Fri, Aug 04, 2017 at 02:09:14AM -0700, jianzh...@gmail.com wrote:

> Hey Guys,
>
> I'm in trouble in the same issue. My code as the following:
>
> test.go
> ```go
> name := []string{"gpu0", "gpu1", "gpu2", "gpu3"}
> matrix := [3][3]int{{1, 0, 0}, {3, 3, 0}, {3, 3, 2}}
>
> C.test_settopologyresource(mod, C.CString("node1"), C.int(2),
> (**C.char)(unsafe.Pointer(&name[0])),
> (**C.int)(unsafe.Pointer(&matrix[0][0])))
> ```

I'm afraid that's not going to work: the elements of the "name" slice
are strings, and they can't be coersed to *C.char because a Go string is
internally a struct consisting of a pointer and a length.

So I think you'd need to make a "clone" data struct -- something like
this:

cnames := make([]*C.Char, len(names))
for i, name := range names {
cnames[i] = C.CString(name)
}

and then destroy those objects after returning from the C side:

for _, p := range cnames {
C.free(p)
}

[...]

[jianzh...@gmail.com]

Thanks your reply, I also used this way, but it still not work. code as the following.

gonameunits := []string{"gpu0", "gpu1", "gpu2", "gpu3"}

   nameunits := make([]*C.char, len(gonameunits))

   for i, _ := range gonameunits {

       nameunits[i] = C.CString(gonameunits[i])

       defer C.free(unsafe.Pointer(nameunits[i]))

   }

   fmt.Println("nameunits:", nameunits)

    golevelmatrix := [][]int{{1}, {3, 3}, {3, 3, 2}}

   levelmatrix := make([][]C.int, len(golevelmatrix))

   for i, _ := range golevelmatrix {

       levelmatrix[i] = make([]C.int, len(golevelmatrix[i]))

       for j, _ := range golevelmatrix[i] {

           levelmatrix[i][j] = C.int(golevelmatrix[i][j])

       }

   }

   fmt.Println("levelmatrix:", levelmatrix)

    C.test_settopologyresource(mod, C.CString("node1"), C.int(2), (**C.char)(unsafe.Pointer(&nameunits[0])), (**C.int)(unsafe.Pointer(&levelmatrix[0][0])))

在 2017年8月4日星期五 UTC+8下午8:09:34，Konstantin Khomoutov写道：

[Konstantin Khomoutov]

On Sun, Aug 06, 2017 at 07:08:03PM -0700, jianzh...@gmail.com wrote:

> Thanks your reply, I also used this way, but it still not work. code as the
> following.
>
> gonameunits := []string{"gpu0", "gpu1", "gpu2", "gpu3"}
> nameunits := make([]*C.char, len(gonameunits))
> for i, _ := range gonameunits {
> nameunits[i] = C.CString(gonameunits[i])
> defer C.free(unsafe.Pointer(nameunits[i]))
> }
> fmt.Println("nameunits:", nameunits)

This looks correct...

> golevelmatrix := [][]int{{1}, {3, 3}, {3, 3, 2}}
> levelmatrix := make([][]C.int, len(golevelmatrix))
> for i, _ := range golevelmatrix {
> levelmatrix[i] = make([]C.int, len(golevelmatrix[i]))
> for j, _ := range golevelmatrix[i] {
> levelmatrix[i][j] = C.int(golevelmatrix[i][j])
> }
> }

...but this is not: remember that in Go, the type []T denotes *a slice* of
elements of type T. A slice is a dynamic construct which is a view into
underlying array, and as such it itself consists of a pointer into that
array's contents, the current length of the slice and the capacity of
it.

> fmt.Println("levelmatrix:", levelmatrix)
>
> C.test_settopologyresource(mod, C.CString("node1"), C.int(2), (**C.char
> )(unsafe.Pointer(&nameunits[0])), (**C.int)(unsafe.Pointer(&levelmatrix[0][0
> ])))

That thing about slices means that passing a pointer to a single slice
element to C is OK; passing a pointer to an Nth element of a slice to C
and expecting it to access it and the elements following it up to the
slice's length is also OK but interpreting a slice of slices (which
[][]*C.int really is) is wrong: it's not represented by a contiguous
array: it's a slice of N independent slices -- each backed by its
independent array.

So what you really want is something like:

1. Know the dimensions of your matrix (let's denote them by N and M).
2. Allocate a slice of the length N*M.
3. Fill it up with the values from your source slice of slices.
4. Pass the pointer to the array to C.

Like this (untested, may not even compile):

n := len(golevelmatrix)
m := 0
for _, row := range golevelmatrix {
if len(row) > m {
m = len(row)
}
}

a := make([]int, n*m)
i := 0
for _, row := range golevelmatrix {
for j, v := range row {
a[i*n+j] = v
}
i++
}

C.my_func(&a[0])

[jianzh...@gmail.com]

Thank you very much for your patience and help. I got it and will try it later. :)

在 2017年8月7日星期一 UTC+8下午3:52:43，Konstantin Khomoutov写道：

[Konstantin Khomoutov]

On Mon, Aug 07, 2017 at 06:25:48AM -0700, jianzh...@gmail.com wrote:

> Thank you very much for your patience and help. I got it and will try it
> later. :)

Glad to help!

> > > golevelmatrix := [][]int{{1}, {3, 3}, {3, 3, 2}}
> > > levelmatrix := make([][]C.int, len(golevelmatrix))
> > > for i, _ := range golevelmatrix {
> > > levelmatrix[i] = make([]C.int, len(golevelmatrix[i]))
> > > for j, _ := range golevelmatrix[i] {
> > > levelmatrix[i][j] = C.int(golevelmatrix[i][j])
> > > }
> > > }

[...]

I'd like to reiterate more precisely: your client is expecting a
multi-dimensional array which, in C, would be a contiguous region of
memory. The Go's [][]C.int is a slice of individual []C.int slices.
Hence when you pass &levelmatrix[0][0] to your C client, it receives the
address of the first element of the first slice, and the only valid
region of memory to access via that pointer is that element and all the
element following it up to (but not including) the length of the first slice.

As soon as the C client attempts to access any memory region other than
that, it may read/write random memory and even invalid memory (at the
addressed which are not allocated/mapped) -- in which case you get
SIGSEGV or the like.

[jianzh...@gmail.com]

Thanks! As my example on the above, I have to change the C client (change `int** _levelmatrix` argument to `int* _levelmatrix`), right? 

在 2017年8月7日星期一 UTC+8下午10:14:44，Konstantin Khomoutov写道：

[jianzh...@gmail.com]

But, if I can not change the C client, how to implement it? the C client as the following. As described on the above, I have to change the ` int** _levelmatrix` argument, right?

bool test_settopologyresource(bsagomodule* _self, char* _nodename,

       int _resourceindex, char** _nameunits, int** _levelmatrix) {

   printf("\n set topology resource-> node:%s, resource index:%d", _nodename, _resourceindex);

   bool result = settopologyresource(_self->client, _nodename, _resourceindex, _nameunits, _levelmatrix);

   return result;

}

在 2017年8月7日星期一 UTC+8下午10:14:44，Konstantin Khomoutov写道：

On Mon, Aug 07, 2017 at 06:25:48AM -0700, jianzh...@gmail.com wrote:

[Tamás Gulácsi]

2017. augusztus 8., kedd 5:20:32 UTC+2 időpontban Jian Zhang a következőt írta:

But, if I can not change the C client, how to implement it? the C client as the following. As described on the above, I have to change the ` int** _levelmatrix` argument, right?

bool test_settopologyresource(bsagomodule* _self, char* _nodename,

       int _resourceindex, char** _nameunits, int** _levelmatrix) {

   printf("\n set topology resource-> node:%s, resource index:%d", _nodename, _resourceindex);

   bool result = settopologyresource(_self->client, _nodename, _resourceindex, _nameunits, _levelmatrix);

   return result;

}

So, your C client wants a 2d slice as a C "array of pointers to arrays".

In the Go side, you have a "slice of slices", where a slice is a struct of length, capacity, and an array of the values. That's why you can pass the array pointer as the pointer to the first element.

So you have to build an array of pointers to arrays:

    golevelmatrix := [][]C.int{{1}, {3, 3}, {3, 3, 2}}

    levelmatrix := make([]*C.int, len(golevelmatrix))

    for i, vv := range golevelmatrix {

        levelmatrix[i] = &vv[0]

    }

    C.test_settopologyresource(..., (**C.int)(unsafe.Pointer(&levelmatrix[0])))

[Konstantin Khomoutov]

On Mon, Aug 07, 2017 at 08:20:01PM -0700, jianzh...@gmail.com wrote:

[...]

> > > > > golevelmatrix := [][]int{{1}, {3, 3}, {3, 3, 2}}
> > > > > levelmatrix := make([][]C.int, len(golevelmatrix))
> > > > > for i, _ := range golevelmatrix {
> > > > > levelmatrix[i] = make([]C.int, len(golevelmatrix[i]))
> > > > > for j, _ := range golevelmatrix[i] {
> > > > > levelmatrix[i][j] = C.int(golevelmatrix[i][j])
> > > > > }
> > > > > }
> > [...]
> >
> > I'd like to reiterate more precisely: your client is expecting a
> > multi-dimensional array which, in C, would be a contiguous region of
> > memory. The Go's [][]C.int is a slice of individual []C.int slices.
> > Hence when you pass &levelmatrix[0][0] to your C client, it receives the
> > address of the first element of the first slice, and the only valid
> > region of memory to access via that pointer is that element and all the
> > element following it up to (but not including) the length of the first
> > slice.

[...]

> But, if I can not change the C client, how to implement it? the C client as
> the following. As described on the above, I have to change the ` int**
> _levelmatrix` argument, right?
> bool test_settopologyresource(bsagomodule* _self, char* _nodename,
> int _resourceindex, char** _nameunits, int** _levelmatrix) {
> printf("\n set topology resource-> node:%s, resource index:%d", _nodename
> , _resourceindex);
> bool result = settopologyresource(_self->client, _nodename,
> _resourceindex, _nameunits, _levelmatrix);
> return result;
> }

Ah, I see.

So, do I understand correctly that your C side is actually expecting
an "array of arrays" (or "jagged array" as some call them)?
That would explain the int** type your function expects.

If yes, this is doable without changing the C side -- by C.malloc()'ing
each member array and storing the pointer to it in the main array:

package main

import "unsafe"

/*
extern int crunch_matrix(int **a);
*/
import "C"

func main() {

golevelmatrix := [][]int{{1}, {3, 3}, {3, 3, 2}}

matrix := make([]*C.int, len(golevelmatrix))

for i, row := range golevelmatrix {
p := (*C.int)(C.malloc(C.size_t(C.sizeof_int * len(row))))
matrix[i] = p

pa := (*[1 << 30]C.int)(unsafe.Pointer(p))

for j, v := range row {

(*pa)[j] = C.int(v)
}
}
C.crunch_matrix(&matrix[0])
}

Here, for each row of the source matrix we allocate an array of C.int of
the source row's size, and place a pointer to it into the destination
slice.

To fill the allocated row array with the source values without jumping
through the hoops of direct pointer arithmetics, we type-convert the
value of "p" (which has type *C.int) to the type *[1 << 30]C.int which
is a pointer to a very long array of C.int (read more on this in the
section "Turning C arrays into Go slices" in [1]).

Note two problems with this approach.

First, I made no effort to free the memory obtained by calls to
C.malloc(). That's mostly for brewity, but see below.

Second, as the code stands now, as soon as we allocate an array for the
target row, the information of its size is lost. We can safely pass the
pointer to it to C.free (because C.malloc stores a special block of
information along with each block it allocates which contains the length
of that block) but I fail to see how your C side is aware of the length
of each row array. On the Go side, the matrix is a slice of slices, and
since a Go slice is fully self-aware, that's OK but in C, an array is
merely a pointer, and it carries no information of the length of the
memory block pointed to by that pointer.

Hence, unless I fail to spot where you encode this information in
the arguments supplied to test_settopologyresource(), this might
indicate a problem with your design of your C side: you might need to
either use a custom type for the rows of your C matrix (a struct
containing a pointer and a size, for instance) or operate on full square
matrices as in my first example.

1. https://github.com/golang/go/wiki/cgo

[jianzh...@gmail.com]

Awesome! Thanks again for your kindly help. :) And I updated the code, it works! 

在 2017年8月8日星期二 UTC+8下午3:51:10，Konstantin Khomoutov写道：