Wayland: cgo, function pointers and *[0]uint8

[Andrew Gallant]

Hello all,

I'm starting work on getting Go able to interact with Wayland
compositors, and I'm running into trouble with a simple wrapping of
libwayland-client. (I mean it only as a toy demo, so I can get my feet
wet.)

Basically, the Wayland API is full of function pointers and uses
callbacks heavily. This poses a problem with cgo because it's
difficult to switch between C and Go function values. I want to use
something like [1] to work around it. It isn't pretty, but it should
be able to work. (I know I'll get hosed if Wayland uses multiple
threads, but as of right now, Wayland doesn't use multiple threads.
Besides, are there any other choices?)

Now, for my real problem. One of the central structs used in Wayland
does not seem to be translatable to a Go type, and I'm not sure why.
Here is a minimal test case [2] that demonstrates my problem (you'll
need the Wayland client libraries installed to run, but I've hopefully
included enough so that you don't have to). The output of [2] is
simply: "Type *[0]uint8". Also, the output of `go tool cgo --godefs`
is here [3].

Obviously, the type of "*[0]uint8" is prohibitory. But is there a way
to work around this?

What I was thinking was that there was something about the wl_display
struct that cannot be translated to Go. So to inspect more closely, I
copied out the wl_display struct (and all of its necessary ancillary
types) and created an isolated example (that does not require Wayland
to run) to see if I could reproduce the *[0]uint8 type. [4]
Interestingly, the output of this program is correct: "Type:
*main._Ctype_struct_wl_display". Inspecting further, here is the
output of `go tool cgo --godefs` [5]. Aha! Notice that the "Update"
and "Global_handler" members have type "*[0]byte". They are function
pointers; could that be why they are getting bogus types?

Can anyone explain what is going on here? Why does the entire
wl_display struct get chopped to *[0]uint8 when wrapping
libwayland-client, but not when the type is reproduced exactly outside
of the Wayland libs? Is it possible to work with *[0]uint8 values?

It would be even better if there is a way to work around this.

(Also, cgo looks nearly required to proceed forward with Wayland and
Go at this point. Otherwise I'd happily do this in pure Go. Namely,
EGL itself links to libwayland-client and therefore needs
libwayland-client values to work. It's possible to do a pure Go
implementation, but it would have to do software rendering. Of course,
one could write EGL in Go, I guess........)

[1] - http://stackoverflow.com/a/6147097/619216
[2] - http://pastebin.com/neHSyTSc
[3] - http://pastebin.com/9TfdeE2S
[4] - http://pastebin.com/ZYJvvfBu
[5] - http://pastebin.com/vS53fsYW

- Andrew

[Andrew Gallant]

While I haven't made any progress on the problem I've described, I
have managed to get an extremely simple Wayland client running. (I did
this by not peeking into Wayland structs and not exporting any Go
functions with Wayland structs in the parameter list.) It's pretty
much the ugliest code ever, but it does work with Weston.

It's go gettable, but you'll need to have Wayland installed.

https://github.com/BurntSushi/go-wayland-simple-shm

- Andrew

[kortschak]

I only started using cgo last week, so take anything I say with a grain of salt, but the way I interpret *[0]uint8 is as void* and this has worked for me. See here: https://github.com/kortschak/boom/blob/master/boom.go (more so in the earlier commits when I had less of a clue about cgo types, but still as a void* to a fn).

Dan

[Andrew Gallant]

> I only started using cgo last week, so take anything I say with a grain of
> salt, but the way I interpret *[0]uint8 is as void* and this has worked for
> me.

I figured as much. The real problem here is that an entire C struct
(which contains other structs, ints and a couple of function pointers)
is translated entirely to just a simple *[0]uint8 (or sometimes a
*[0]byte). When this happens, I don't know how to deal with it in Go.
I can still pass it as a value to other C functions, but I can't
access its members when in Go. Moreover, since it gets translated to a
*[0]uint8, any function that has a parameter with a
*C.struct_wl_display type cannot be exported. (Which is a problem
because Wayland makes heavy use of callbacks.)

The other puzzling part (to me) is that this only happens when I use
the Wayland libraries directly, but not when I manually copy the
Wayland types right into the Go source file.

P.S. kortschak - your biogo library is awesome. I'm using it in a
research project right now :-)

- Andrew

[kortschak]

You can unsafe type convert to get the fields you need.

I'd really like to hear from people who know why *[0]byte is used here (if I'm correct in my interpretation above) rather than unsafe.Pointer (maybe just because it means unsafe isn't needed to be imported?).

Thanks for the feedback on biogo. You may be interested in boom if you're doing bioinfo work. Also, please have a look at the API for the exp packages - they are way too bloated for my liking, but I'm too close to them to really be sure, so feedback there would be appreciated.

Dan

[Andrew Gallant]

> You can unsafe type convert to get the fields you need.

How? I understand how to do it with a Go type:

(*someGoType)(someUnsafePointer)

but in my case, "someGoType" is actually a C type whose translation to
Go is simply *[0]byte. So that

(*C.struct_wl_display)(someStruct)

won't do anything. In fact, this is the important bit in my first
pasty:

var disp *C.struct_wl_display
fmt.Printf("Type: %T\n", disp)

Where the output is "*uint8[0]". The godefs are:

var disp *[0]byte
fmt.Printf("Type: %T\n", disp)

The key here is that I'm already telling Go that 'disp' is a
*C.struct_wl_display. So what am I going to use to convert a *uint8[0]
to a *C.struct_wl_display if *C.struct_wl_display is itself a
*uint8[0]?

Using something like 'type blahblah C.struct_wl_display' doesn't work
either, and is translated to 'type blahblah [0]byte'.

> Thanks for the feedback on biogo. You may be interested in boom if you're
> doing bioinfo work. Also, please have a look at the API for the exp
> packages - they are way too bloated for my liking, but I'm too close to
> them to really be sure, so feedback there would be appreciated.

I'm going to be diving back into my research project tomorrow, so
hopefully I'll get chance to take a look at the exp packages. I'll see
if I can come up with any feedback. (The piece I'm interested in now
is your Smith-Waterman alignment implementation, which is beautifully
written.)

- Andrew

[kortschak]

I don't know if this is idiomatic, but I have been wrapping C types in Go structs. These print fine, and so must be reflected on properly. I guess it depends on what you want to access the struct for, you can directly access the fields without doing that.

When reading the exp stuff, have in mind that in the near future all seq objects will satisfy a Feature interface (currently this is defined in a local branch to be something like this https://gist.github.com/2913911).

The SW implementation is really a teaching example, it's horribly inefficient compared to the optimised implementations that are out in the wild, but is is hopefully clear.
 

- Andrew

[Steven Blenkinsop]

I haven't used cgo, but I can tell you that the struct wl_display is an opaque type. It is declared in wayland-client-protocol.h and defined in wayland-client.c. This is why it's being translated as [0]uint8: at compile time, you don't actually know what it is. I think the solution is to copy the definition of struct wl_display as a new type into your own C code, and convert the pointers yourself. You're working against the library here because it clearly intends for you not to use (read "depend on") the fields. This is true whether you're working from C or Go, if I'm not mistaken.