bug in cgo with C callbacks?

[GreatOdinsRaven]

I've been playing around with Cgo and came across a strange crash. The use is case extremely simple and I have no idea what I'm doing wrong. It really looks like a bug in cgo to me but I wanted to confirm that I'm not missing something simple. 

I'm on 64-bit Darwin, 10.8.3, 64-bit, Go 1.1 compiled from source.  

I have the following Go code:

package main

import (

"fmt"

"unsafe"

)

/*

#include <stdlib.h>

void my_printf(char* str);

*/

import "C"

//export hello_from_c

func hello_from_c(s string, i int) {

fmt.Printf("From C: %s, %d\n", s, i)

}

func main() {

cstr := C.CString("GoTest")

defer C.free(unsafe.Pointer(cstr))

C.my_printf(cstr)

}

It's purpose is to pass a Go string to C land, where it is printed. The C function then passes back a C-string and an int which are printed. Simple enough. 

This is the C code:

#include <stdio.h>

#include <stdlib.h>

typedef signed char GoInt8;

typedef unsigned char GoUint8;

typedef short GoInt16;

typedef unsigned short GoUint16;

typedef int GoInt32;

typedef unsigned int GoUint32;

typedef long long GoInt64;

typedef unsigned long long GoUint64;

typedef GoInt64 GoInt;

typedef GoUint64 GoUint;

typedef __SIZE_TYPE__ GoUintptr;

typedef float GoFloat32;

typedef double GoFloat64;

typedef __complex float GoComplex64;

typedef __complex double GoComplex128;

typedef struct { char *p; int n; } GoString;

typedef void *GoMap;

typedef void *GoChan;

typedef struct { void *t; void *v; } GoInterface;

typedef struct { void *data; int len; int cap; } GoSlice;

extern void hello_from_c(GoString p0, GoInt p1);

void my_printf(char *s){

printf("From Go: %s\n", s);

GoString d;

d.p = "C Test";

d.n = 7;

// printf("C STRING: %s\n", d.p);

// fflush(stdout);

hello_from_c(d, 42);

}

After I run it:

From Go: GoTest

runtime: out of memory: cannot allocate 140733193453568-byte block (1048576 in use)

fatal error: out of memory

goroutine 1 [running]:

[fp=0x41d4700] runtime.throw(0x410b465)

/Users/dimitarvelitchkov/Documents/Projects/GoProjects/GoSrc/go/src/pkg/runtime/panic.c:473 +0x67

[fp=0x41d4758] runtime.mallocgc(0x7fff0000000f, 0x100000001, 0x1)

/Users/dimitarvelitchkov/Documents/Projects/GoProjects/GoSrc/go/src/pkg/runtime/zmalloc_darwin_amd64.c:60 +0x316

[fp=0x41d4790] makeslice1(0x40807c0, 0x8, 0x7fff0000000f, 0x41d4850)

/Users/dimitarvelitchkov/Documents/Projects/GoProjects/GoSrc/go/src/pkg/runtime/slice.c:61 +0x89

[fp=0x41d47c0] growslice1(0x40807c0, 0xc200000018, 0x8, 0x8, 0x7fff0000000f, ...)

/Users/dimitarvelitchkov/Documents/Projects/GoProjects/GoSrc/go/src/pkg/runtime/slice.c:230 +0x56

[fp=0x41d4820] runtime.appendstr()

/Users/dimitarvelitchkov/Documents/Projects/GoProjects/GoSrc/go/src/pkg/runtime/slice.c:147 +0x79

[fp=0x41d48c0] fmt.(*fmt).padString(0xc20005b128, 0x4067e1d, 0x7fff00000007)

/Users/dimitarvelitchkov/Documents/Projects/GoProjects/GoSrc/go/src/pkg/fmt/format.go:129 +0x1ba

[fp=0x41d48f0] fmt.(*fmt).fmt_s(0xc20005b128, 0x4067e1d, 0x7fff00000007)

/Users/dimitarvelitchkov/Documents/Projects/GoProjects/GoSrc/go/src/pkg/fmt/format.go:272 +0x5b

[fp=0x41d4920] fmt.(*pp).fmtString(0xc20005b0d0, 0x4067e1d, 0x7fff00000007, 0x73)

/Users/dimitarvelitchkov/Documents/Projects/GoProjects/GoSrc/go/src/pkg/fmt/print.go:537 +0xb3

[fp=0x41d4980] fmt.(*pp).printField(0xc20005b0d0, 0x4080340, 0xc200039160, 0x73, 0x0, ...)

/Users/dimitarvelitchkov/Documents/Projects/GoProjects/GoSrc/go/src/pkg/fmt/print.go:799 +0x9b6

[fp=0x41d4d00] fmt.(*pp).doPrintf(0xc20005b0d0, 0x40b05b0, 0xf, 0x41d4e08, 0x2, ...)

/Users/dimitarvelitchkov/Documents/Projects/GoProjects/GoSrc/go/src/pkg/fmt/print.go:1111 +0x1276

[fp=0x41d4d50] fmt.Fprintf(0xc200058150, 0xc200000008, 0x40b05b0, 0xf, 0x41d4e08, ...)

/Users/dimitarvelitchkov/Documents/Projects/GoProjects/GoSrc/go/src/pkg/fmt/print.go:214 +0x80

[fp=0x41d4da8] fmt.Printf(0x40b05b0, 0xf, 0x41d4e08, 0x2, 0x2, ...)

/Users/dimitarvelitchkov/Documents/Projects/GoProjects/GoSrc/go/src/pkg/fmt/print.go:223 +0x8c

[fp=0x41d4e30] main.hello_from_c(0x4067e1d, 0x7fff00000007, 0x2a)

play/_obj/play.cgo1.go:16 +0xdd

----- stack segment boundary -----

[fp=0x41d4ee0] runtime.cgocallbackg(0x7fff5fbff870, 0x7fff5fbff8f8, 0x18)

/Users/dimitarvelitchkov/Documents/Projects/GoProjects/GoSrc/go/src/pkg/runtime/cgocall.c:260 +0x118

[fp=0x41d4f00] runtime.cgocallback_gofunc()

/Users/dimitarvelitchkov/Documents/Projects/GoProjects/GoSrc/go/src/pkg/runtime/asm_amd64.s:635 +0x71

[fp=0x41d4f08] return()

/Users/dimitarvelitchkov/Documents/Projects/GoProjects/GoSrc/go/src/pkg/runtime/asm_amd64.s:508

[fp=0x41d4f50] runtime.cgocall(0x4000c20, 0x41d4f68)

/Users/dimitarvelitchkov/Documents/Projects/GoProjects/GoSrc/go/src/pkg/runtime/cgocall.c:162 +0x128

[fp=0x41d4f68] main._Cfunc_my_printf(0x4400000)

play/_obj/_cgo_defun.c:52 +0x2f

[fp=0x41d4f90] main.main()

play/_obj/play.cgo1.go:28 +0x64

[fp=0x41d4fb8] runtime.main()

Here's the wacky part. If I uncomment the two lines currently commented out in the C file....it works. Just by printing the raw char* to stdout somehow causes the whole thing to work. Something tells me memory is being prematurely deallocated/corrupted but I don't understand where, how, and why. 

[Dmitry Vyukov]

I don't know where you get it, but n is not int in GoString:

typedef struct { char *p; int n; } GoString;

You should use something like size_t.

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[GreatOdinsRaven]

Thanks for looking into this

The GoString typedef is copied verbatim from _cgo_export.h. I copy-pasted all typedefs to get the .c code to compile. Couldn't just #include the export header because it wasn't generated at the time the .c code was being compiled. I could be doing something wrong there, I don't know. 

Here's the contents of the export header:

dimitars-imac:_obj dimitarvelitchkov$ cat _cgo_export.h

/* Created by cgo - DO NOT EDIT. */

#line 8 "play.go"

#include <stdlib.h>

void my_printf(char* str);

typedef signed char GoInt8;

typedef unsigned char GoUint8;

typedef short GoInt16;

typedef unsigned short GoUint16;

typedef int GoInt32;

typedef unsigned int GoUint32;

typedef long long GoInt64;

typedef unsigned long long GoUint64;

typedef GoInt64 GoInt;

typedef GoUint64 GoUint;

typedef __SIZE_TYPE__ GoUintptr;

typedef float GoFloat32;

typedef double GoFloat64;

typedef __complex float GoComplex64;

typedef __complex double GoComplex128;

typedef struct { char *p; int n; } GoString;

typedef void *GoMap;

typedef void *GoChan;

typedef struct { void *t; void *v; } GoInterface;

typedef struct { void *data; int len; int cap; } GoSlice;

extern void hello_from_c(GoString p0, GoInt p1);

I followed your suggestion and changed the int to size_t, but that didn't help. I still get it to crash. And again, uncommenting the printf + fflush   makes the program run (the real mystery to me). 

If the int inside the GoString typedef needs to be a size_t, it sounds like a bug in the code generation tool. But I still think there's a bug somewhere else too, unrelated to that. 

Thanks!

[minux]

On Thu, May 23, 2013 at 10:08 PM, GreatOdinsRaven <dimiter....@gmail.com> wrote:

Thanks for looking into this

The GoString typedef is copied verbatim from _cgo_export.h. I copy-pasted all typedefs to get the .c code to compile. Couldn't just #include the export header because it wasn't generated at the time the .c code was being compiled. I could be doing something wrong there, I don't know. 

the correct way is still to include _cgo_export.h, see $GOROOT/misc/cgo/test/callback_c.c

for example.

are you using the go tool (instead invoking the cgo command directly) to build the program?

btw, the definition of GoSlice and GoString should use type GoInt for n, len and cap, i don't

know why your struct definitions are incorrect. what version of go are you using?

[GreatOdinsRaven]

I'm using the go tool, yes. "go build play" is how I invoke it. All my files are placed under $GOPATH/src/play. If I compile with "go", it doesn't keep the "_obj" directory, so  I compiled with "go tool cgo" to try to debug it. 

I thought including the _cgo_export.h was the way to go as well, but it was a chicken-and-egg issue with the file not being generated (yet) at the time the .c file was being compiled. I must be doing something very wrong with the "go build" invocation then. 

I'm using Go 1.1, compiled from source. Since you're saying there are even more typedef's that are wrong, this looks more and more like some configuration issue specific to my system. I'm going to try the same test tonight with Go 1.1 from the binary distribution. 

Thanks!

[minux]

On Thu, May 23, 2013 at 10:50 PM, GreatOdinsRaven <dimiter....@gmail.com> wrote:

I'm using the go tool, yes. "go build play" is how I invoke it. All my files are placed under $GOPATH/src/play. If I compile with "go", it doesn't keep the "_obj" directory, so  I compiled with "go tool cgo" to try to debug it. 

I thought including the _cgo_export.h was the way to go as well, but it was a chicken-and-egg issue with the file not being generated (yet) at the time the .c file was being compiled. I must be doing something very wrong with the "go build" invocation then.

this is strange. I suggest you take a look at misc/cgo/test for extensive cgo examples.

_cgo_export.h will be generated before any of the C source code is compiled.

I'm using Go 1.1, compiled from source. Since you're saying there are even more typedef's that are wrong, this looks more and more like some configuration issue specific to my system. I'm going to try the same test tonight with Go 1.1 from the binary distribution.

let me give your an example session running your source code with minimum editing:

$ mkdir -p  /tmp/gopath/src

$ cd /tmp/gopath/src

$ cat > main.go <<EOF

package main

import (

        "fmt"

        "unsafe"

)

/*

        #include <stdlib.h>

        void my_printf(char* str);

*/

import "C"

//export hello_from_c

func hello_from_c(s string, i int) {

        fmt.Printf("From C: %s, %d\n", s, i)

}

func main() {

        cstr := C.CString("GoTest")

        defer C.free(unsafe.Pointer(cstr))

        C.my_printf(cstr)

}

EOF

$ cat > part.c <<EOF

#include <_cgo_export.h>

#include <stdio.h>

void my_printf(char *s){

        printf("From Go: %s\n", s);

        GoString d;

        d.p = "C Test";

        d.n = 7;

        // printf("C STRING: %s\n", d.p);

        // fflush(stdout);

        hello_from_c(d, 42);

}

EOF

$ GOPATH=/tmp/gopath go install cgotest

$ /tmp/gopath/bin/cgotest

From Go: GoTest

From C: C Test, 42

[andrey mirtchovski]

minux,

following your steps exactly, except for:

$ mkdir -p /tmp/gopath/src/cgotest
$ cd /tmp/gopath/src/cgotest

I get:

$ /tmp/gopath/bin/cgotest
From Go: GoTest

runtime: out of memory: cannot allocate 140733193453568-byte block
(1048576 in use)
fatal error: out of memory

[snip, full dump available: http://play.golang.org/p/QsokZKHkRr]

I notice that if I make the following modification,
http://play.golang.org/p/J6fUEF4E_9, then the function works just
fine, but if I replace strlen(t) on line 10 with the value it returns,
6, the resulting program crashes again. putting a panic() inside
hello_from_c in main.go gives this interesting observation:

with strlen(t):
main.hello_from_c(0x40680b5, 0x6, 0x2a)
cgotest/_obj/main.cgo1.go:17 +0x119

with "6" instead:
[fp=0x41d1e30] main.hello_from_c(0x40680bd, 0x7fff00000006, 0x2a)
cgotest/_obj/main.cgo1.go:16 +0xdd

notice the value passed for 'i'.

[Ian Lance Taylor]

I see the bug. _cgo_export.c looks like this:

void hello_from_c(GoString p0, GoInt p1)

{
struct {
GoString p0;
GoInt p1;
} __attribute__((packed)) a;
a.p0 = p0;
a.p1 = p1;
crosscall2(_cgoexp_44bb5d457081_hello_from_c, &a, 24);
}

_cgo_export.h has this:

typedef struct { char *p; int n; } GoString;

This means that the generated hello_from_c code is going to convert
from GoInt to int when setting the string length of p0. That will
break if it so happens that the upper 32 bits of the string length are
not already zero.

Please open an issue for this. Thanks.

Ian

[andrey mirtchovski]

I've created issue 5548:

[andrey mirtchovski]

sorry, command-enter strikes again.

here's the link that didn't post before:
https://code.google.com/p/go/issues/detail?id=5548