[go-nuts] SWIG interface for callbacks

[Ian Lance Taylor]

As I've mentioned before, I'm adding support for Go to SWIG. This is
another question about suggestions for how to best use it.

SWIG is a way to permit other languages to call C++. One of SWIG's
features is permitting an object in the other language to define
virtual functions which may be called from C++ routines which know
nothing about the other language. That is, given a C++ class like

class C { public: virtual void callback(); };

a C++ function like

void fn(C* c) { c->callback(); }

can wind up calling a function in some other language. This is a
convenient feature for C++ programs which communicate via callbacks
implemented as virtual functions.

The mechanics of this are complex, but for languages like Java and
Python it is conceptually straightforward. SWIG arranges matters such
that the Java code can write

class JavaC extends C { public void run() { ... } }

or Python code can write

class PyC(package.C)
def run(self):
...

An instance of those classes can be created and passed back to some
C++ function.

This conceptually simple interface is not available in Go, because Go
does not have C++ style class inheritance. In order to make this
work, it's necessary to be able to write a function whose argument can
be either a C or a GoC. In Java or Python you get that via
inheritance, but in Go it means that the argument must be an
interface. That in turns means that there must be some other type,
let's say a struct, which implements the interface. A struct can
inherit methods from an anonymous embedded struct, but those methods
are not virtual as they are in C++; if a method of the embedded struct
calls another method, it will call the method on the embedded struct,
not the method defined for the enclosing type. This is the key
difference between Go-style inheritance and C++-style inheritance, and
it means that there is no natural way for one style to represent the
other.

So, this is what I have come up with. SWIG calls these feature of
using callbacks the "director" feature and requires you to enable it
for specific classes using %feature(director). Therefore, when I see
this, I have SWIG emit things like this:

type C interface {
Run()
}

type _swig_DirectorC struct { }
// Define methods so that _swig_DirectorC satisfies the interface C

func NewDirectorC(v interface{}) *_swig_DirectorC

The name "_swig_DirectorC" is unimportant as the user of the package
will never see it. The idea here is that the caller will define a
type with methods, and will pass an instance of that type to
NewDirectorC. The caller will get back a pointer which satisfies the
interface C, and which can be passed to C++ as though it were a C*.
When the C++ code calls a method on the C* value, it will call back
into Go which will check whether the value implements that method. If
it does, it will be called; otherwise, the C++ parent class's method
will be called.

In other words:

type GoC struct { }
func (p *GoC) Run() { ... }

func fn() {
c := cxx_package.NewDirectorC(&GoC{})
// These call C++ functions
cxx_package.RegisterCallback(c)
cxx_package.RunCallbacks()
}

When the RunCallbacks function invokes the run method on the
registered callback it will wind up calling GoC's Run method.

That was all rather longwinded, but my question is whether anybody
sees a simpler approach to permitting this kind of operation. Is
there a way to implement this with some more direct form of
inheritance, rather than introducing a NewDirectorC function?

Ian


--
Subscription settings: http://groups.google.com/group/golang-nuts/subscribe?hl=en

[Hans Stimer]

I can't think of a simpler approach.

Is it possible that we could get something similar for cgo?

btw - the cgo docs are kind of sparse and they don't seem to sync up with how it is used. For instance, the docs say that a cgo file is part of the pseudo package "C". However, it appears from the code that .cgo files are part of whatever package they are in.

[Russ Cox]

> how it is used. For instance, the docs say that a cgo file is part of the
> pseudo package "C". However, it appears from the code that .cgo files are
> part of whatever package they are in.

Where does it say that? I'd like to correct it.

You are right that the cgo inputs (actually .go files)
are part of whatever package they are in; they just
import "C".

Russ

[Hans Stimer]

My mistake, it doesn't says that. 

http://golang.org/cmd/cgo/

However, there appears to be a difference in how the packages use cgo and the cgo examples.

The docs say you must import "C", which isn't imported by any of the standard packages that appear to use cgo. 

The file naming convention appears to be that the go files to be processed by cgo end with .cgo 

Additionally, the docs say that the c include files are specified in comments before the import "C" statement. The .cgo files are just putting the #include statements right after the package declaration. The #include are not commented out.

Finally, the pseudo package "C" isn't documented. From looking at the examples, I see there is at least one function in it: C.CString which appears to create a c string from go string. Are there others?

[Russ Cox]

> The docs say you must import "C", which isn't imported by any of the
> standard packages that appear to use cgo.

Which packages are you referring to?

> The file naming convention appears to be that the go files to be processed
> by cgo end with .cgo
> Additionally, the docs say that the c include files are specified in
> comments before the import "C" statement. The .cgo files are just putting
> the #include statements right after the package declaration. The #include
> are not commented out.

Unfortunately we've used the name cgo for two different things.
The one with the import "C" lines and which is documented by
'godoc cgo' and http://golang.org/cmd/cgo/ is the external one,
the one that we mean when we encourage people to use cgo.
The examples of using this tool are in $GOROOT/misc/cgo/*.

It sounds like you are looking at the files in src/pkg/runtime
that are unfortunately named *.cgo. Those use a different tool
that you might think of as a precursor to the "real" cgo.
Those are for writing the low-level bits of runtime and are
basically C files with some functions written using Go declarations
instead of C declarations. They cannot #include standard system
libraries, only the low-level runtime headers, and they cannot
include Go code - you're forced to write in C. We should really
come up with a different name for those files. That tool is
intentionally undocumented.

If you stick to looking at the examples in misc/cgo I think
you'll find the situation clearer and easier to understand.

The pseudo-package "C" contains whatever C symbols
you would like to use, as possible. It also contains two
pre-defined symbols that you won't find in the C libraries:
CString(s) allocates and returns a new C string that is
a copy of the given Go string, and GoString(s) does the
reverse.

[Sverre Rabbelier]

Heya,

On Sun, Apr 18, 2010 at 22:44, Russ Cox <r...@golang.org> wrote:
> Those use a different tool that you might think of as a

> precursor to the "real" cgo. [...]
> That tool is intentionally undocumented. [...]

The way you describe it makes me wonder why those files are not
rewritten using the "new" cgo? Lack of time, the technical limitations
of the "new" cgo, something else?

--
Cheers,

Sverre Rabbelier

[Russ Cox]

>> Those use a different tool that you might think of as a
>> precursor to the "real" cgo. [...]
>> That tool is intentionally undocumented. [...]
>
> The way you describe it makes me wonder why those files are not
> rewritten using the "new" cgo? Lack of time, the technical limitations
> of the "new" cgo, something else?

They're just different programs serving different needs.
One is for writing Go code to wrap C, and the other is
for writing C code to implement Go.

The real cgo is for writing Go code that wraps external C code
which has been compiled with gcc and is linked in at runtime as
a shared library: think wrapping gmp, gtk, sdl, openssl, etc.
Cgo's role is to let you write Go code that uses C by referring
to names in the pseudo-package you get by doing import "C".

The unfortunately-named cgo2c in the runtime directory is for
writing C code callable from Go to implement basic runtime
operations like allocating data or concatenating strings.
That C code is compiled with 6c, not gcc, and it is linked
statically into the program just like a Go source file would be.
Cgo2c's role is to let the Go compiler emit calls to runtime
functions that happen to be implemented in C. The only
useful thing it does is rewrite the Go-style function declarations
into equivalent C-style ones, inserting appropriate padding so
that they end up with the same memory layout.

Russ

[Sverre Rabbelier]

Heya,

On Sun, Apr 18, 2010 at 23:05, Russ Cox <r...@golang.org> wrote:
> They're just different programs serving different needs.
> One is for writing Go code to wrap C, and the other is
> for writing C code to implement Go.

<rest interesting explanation snipped>

The name "cgo" makes sense for both programs (and their files), thanks
for explaining :).

--
Cheers,

Sverre Rabbelier

[Hans Stimer]

Thank you very much for the clarification.

Is there a way to pass a callback routine to c using cgo?

Do I need to worry about memory being freed while the c routines are referencing it?

Can c return a block of memory to go? And then be type casted into go datatype? And then not cause problems for the gc?

It appears that the c code can call go. Are there any rules about this?

[Russ Cox]

On Sun, Apr 18, 2010 at 14:19, Hans Stimer <hans....@gmail.com> wrote:
> Thank you very much for the clarification.
> Is there a way to pass a callback routine to c using cgo?

Yes. Ian just added support for this.
See misc/cgo/life for an exmaple.

> Do I need to worry about memory being freed while the c routines are
> referencing it?

You should arrange to keep a reference to the memory on the Go side
for as long as it is needed by the C side.

> Can c return a block of memory to go? And then be type casted into go
> datatype? And then not cause problems for the gc?

C can return a block of memory to Go, but that memory will
not be garbage collected: it becomes your job to call C.free
when finished with it. The same applies to memory returned
by C.CString.

> It appears that the c code can call go. Are there any rules about this?

C code can't call Go functions directly except via the mechanism
that misc/cgo/life demonstrates.

[Hans Stimer]

 

> Is there a way to pass a callback routine to c using cgo?

Yes.  Ian just added support for this.
See misc/cgo/life for an exmaple.

I've gone through the life example, and I don't see any go code passing a func to a c routine.

btw - the rest of the example is great. It answers many questions. 

[Ostsol]

On Apr 18, 4:18 pm, Hans Stimer <hans.sti...@gmail.com> wrote:
> I've gone through the life example, and I don't see any go code passing a
> func to a c routine.

You have to use a somewhat indirect approach. cgo creates a wrapper
function for each exported Go function. Unfortunately, there is no
direct way to pass that wrapper to an external C library. One has to
create yet another wrapper to do this. Here is the test program I
created:

//callgo.h

typedef int(*gofunc)(int, int);

extern void call_go(gofunc);

//callgo.c

#include <stdio.h>
#include "callgo.h"

void
call_go(gofunc func) {
printf("Calling Go function\n");
int i = func(3, 4);
printf("Go function returned %d\n", i);
}

callgo is the library that takes a Go function and calls it. Nothing
special.

//funcwrap.h

extern void pass_GoAdd(void);

//funcwrap.c

#include "_cgo_export.h"
#include "callgo.h"

void
pass_GoAdd(void){
return call_go(GoAdd);
}

funcwrap calls the wrapper created by cgo. This is necessary because
the wrapper cgo creates is not directly visible to Go.

//callback.go

package callback

// #include "callgo.h"
// #include "funcwrap.h"
import "C"

func Run() {
C.pass_GoAdd()
}

//export GoAdd
func GoAdd(a, b int) int {
return a + b
}

Here we call our function passing code. Unfortunately, it does not
allow us to pass an arbitrary function, thus requiring yet another
level of indirection in order to pass a function not defined within
one's cgo code. Ugly, yes -- but it works.

Here's the Makefile:

include $(GOROOT)/src/Make.$(GOARCH)

TARG=callback
CGOFILES=callback.go
LDPATH_linux=-Wl,-R,`pwd`
CGO_LDFLAGS=_cgo_export.o callgo.so funcwrap.so $(LDPATH_linux)
CGO_DEPS=_cgo_export.o callgo.so funcwrap.so

CLEANFILES+=main

include $(GOROOT)/src/Make.pkg

callgo.o: callgo.c
gcc -g -c -fPIC callgo.c

callgo.so: callgo.o
gcc -shared -o $@ callgo.o

funcwrap.o: funcwrap.c _cgo_export.h
gcc $(_CGO_CFLAGS_$(GOARCH)) -g -c -fPIC $(CFLAGS) funcwrap.c

funcwrap.so: funcwrap.o
gcc $(_CGO_CFLAGS_$(GOARCH)) -o $@ funcwrap.o $(_CGO_LDFLAGS_$
(GOOS))

main: callgo.so install main.go
$(GC) main.go
$(LD) -o $@ main.$O

-Daniel

[Hans Stimer]

Ah.

Ostsol, thank you for working this out. I never would of guessed that this was the calback mechanism.

Hopefully the new go swig approach will be applied to cgo.

[roger peppe]

On 18 April 2010 22:05, Russ Cox <r...@golang.org> wrote:
> They're just different programs serving different needs.
> One is for writing Go code to wrap C, and the other is
> for writing C code to implement Go.

yes, this confused me. particularly as naming my
cgo file with a ".cgo" extension causes its build to fail.

[Serge Hulne]

On Apr 17, 10:43 pm, Ian Lance Taylor <i...@google.com> wrote:
> As I've mentioned before, I'm adding support for Go toSWIG.  This is

> another question about suggestions for how to best use it.
>

> SWIGis a way to permit other languages to call C++.  One ofSWIG's

> features is permitting an object in the other language to define
> virtual functions which may be called from C++ routines which know
> nothing about the other language.  That is, given a C++ class like
>
> class C { public: virtual void callback(); };
>
> a C++ function like
>
> void fn(C* c) { c->callback(); }
>
> can wind up calling a function in some other language.  This is a
> convenient feature for C++ programs which communicate via callbacks
> implemented as virtual functions.
>
> The mechanics of this are complex, but for languages like Java and

> Python it is conceptually straightforward.  SWIGarranges matters such

> that the Java code can write
>
> class JavaC extends C { public void run() { ... } }
>
> or Python code can write
>
> class PyC(package.C)
>     def run(self):
>         ...
>
> An instance of those classes can be created and passed back to some
> C++ function.
>
> This conceptually simple interface is not available in Go, because Go
> does not have C++ style class inheritance.  In order to make this
> work, it's necessary to be able to write a function whose argument can
> be either a C or a GoC.  In Java or Python you get that via
> inheritance, but in Go it means that the argument must be an
> interface.  That in turns means that there must be some other type,
> let's say a struct, which implements the interface.  A struct can
> inherit methods from an anonymous embedded struct, but those methods
> are not virtual as they are in C++; if a method of the embedded struct
> calls another method, it will call the method on the embedded struct,
> not the method defined for the enclosing type.  This is the key
> difference between Go-style inheritance and C++-style inheritance, and
> it means that there is no natural way for one style to represent the
> other.
>

> So, this is what I have come up with.  SWIGcalls these feature of

> using callbacks the "director" feature and requires you to enable it
> for specific classes using %feature(director).  Therefore, when I see

> this, I haveSWIGemit things like this:

>
> type C interface {
>         Run()
>
> }
>
> type _swig_DirectorC struct { }
> // Define methods so that _swig_DirectorC satisfies the interface C
>
> func NewDirectorC(v interface{}) *_swig_DirectorC
>
> The name "_swig_DirectorC" is unimportant as the user of the package
> will never see it.  The idea here is that the caller will define a
> type with methods, and will pass an instance of that type to
> NewDirectorC.  The caller will get back a pointer which satisfies the
> interface C, and which can be passed to C++ as though it were a C*.
> When the C++ code calls a method on the C* value, it will call back
> into Go which will check whether the value implements that method.  If
> it does, it will be called; otherwise, the C++ parent class's method
> will be called.
>
> In other words:
>
> type GoC struct { }
> func (p *GoC) Run() { ... }
>
> func fn() {
>         c := cxx_package.NewDirectorC(&GoC{})
>         // These call C++ functions

Hi Ian,

Would SWIG be suited to to automatically create bindings between Go
and GTK+ ?

After all if I look at Python, It appears that Python bindings to GTK
+, WxWindows, FLTK, etc ... are generated automatically using SWIG.

Serge.

[Ian Lance Taylor]

Serge Hulne <serge...@gmail.com> writes:

> Would SWIG be suited to to automatically create bindings between Go
> and GTK+ ?
>
> After all if I look at Python, It appears that Python bindings to GTK
> +, WxWindows, FLTK, etc ... are generated automatically using SWIG.

If SWIG works for Python, then it should work for Go.

The SWIG code is not yet finalized, but it is close. It has been
committed to the swig SVN repository, but there is still a bit more
tweaking before it can be considered finished.

Ian

[Serge Hulne]

Hi Ian,

May I ask, if you could notify me when SWIG for Go becomes
operational.

I would like to try to assess how easy (or how complex) it would be to
attempt to generate GTK+ bindings for Go.

However, I would rather prefer to wait until SWIG is ripe enough, so
that said test wouldn't be biased by additional difficulties which
might arise from a SWIG version which wouldn't be reasonably stable.

Serge


On Jun 13, 7:40 am, Ian Lance Taylor <i...@google.com> wrote:

[Ian Lance Taylor]

Serge Hulne <serge...@gmail.com> writes:

> May I ask, if you could notify me when SWIG for Go becomes
> operational.

I will send mail to the golang-nuts list.

> I would like to try to assess how easy (or how complex) it would be to
> attempt to generate GTK+ bindings for Go.
>
> However, I would rather prefer to wait until SWIG is ripe enough, so
> that said test wouldn't be biased by additional difficulties which
> might arise from a SWIG version which wouldn't be reasonably stable.

Oh, you can certainly try it now. In fact, I would encourage you to
do so, and report any problems that you find. The details which
remain are things like the precise names to use in the generated
wrapping code.

Ian

[Serge Hulne]

All right,

Thank you !
I'll give it a go.

Serge.

On Jun 13, 8:11 am, Ian Lance Taylor <i...@google.com> wrote:

[Serge Hulne]

Hi Ian,

Which version of swig support Go ?
(I installed the 2.0 version, but it does not mention Go as a target
language)

Thanks,
Serge

[Serge Hulne]

Hi Ian,

I installed swig with:

svn co https://swig.svn.sourceforge.net/svnroot/swig/trunk swig

It compiled, but when doing the make install, I got:

Installing ccache-swig
Installing /usr/local/bin/ccache-swig
Installing /usr/local/share/man/man1/ccache-swig.1
install: ./ccache-swig.1: No such file or directory
make[1]: *** [install] Error 71

I am using Mac OS X 10.6 (with the XCode code).

Serge



On Jun 13, 10:20 am, Serge Hulne <serge.hu...@gmail.com> wrote:

[Ian Lance Taylor]

Serge Hulne <serge...@gmail.com> writes:

> I installed swig with:
>
> svn co https://swig.svn.sourceforge.net/svnroot/swig/trunk swig
>
> It compiled, but when doing the make install, I got:
>
> Installing ccache-swig
> Installing /usr/local/bin/ccache-swig
> Installing /usr/local/share/man/man1/ccache-swig.1
> install: ./ccache-swig.1: No such file or directory
> make[1]: *** [install] Error 71
>
> I am using Mac OS X 10.6 (with the XCode code).

I don't know why this happens, but I think you can avoid it by using
the --disable-ccache option when you run configure.

Ian