Out-of-band callbacks with cgo

51 views
Skip to first unread message

Gustavo Niemeyer

unread,
Oct 29, 2010, 12:01:12 PM10/29/10
to golang-nuts
Greetings,

I'd like to wrap the Zookeeper [1] C library for Go, and I believe
this is touching on an open problem related to C callbacks.

The thread-safe and recommended way that Zookeeper can be wrapped
spawns a thread in the background to handle incoming events such as
watches on nodes. Ian was telling me that there's no way for that
thread to callback into Go code, though. The potential workaround I
can see is to busy loop in the Go side, and dequeue callbacks when
they are available, but that's of course sub-optimal.

Is there a better way to do this at the moment, and is the solution to
the underlying problem of being unable to callback from C in the
horizon?

Thanks,

[1] http://hadoop.apache.org/zookeeper/

--
Gustavo Niemeyer
http://niemeyer.net
http://niemeyer.net/blog
http://niemeyer.net/twitter

Florian Weimer

unread,
Oct 29, 2010, 5:27:59 PM10/29/10
to Gustavo Niemeyer, golang-nuts
* Gustavo Niemeyer:

> The thread-safe and recommended way that Zookeeper can be wrapped
> spawns a thread in the background to handle incoming events such as
> watches on nodes. Ian was telling me that there's no way for that
> thread to callback into Go code, though. The potential workaround I
> can see is to busy loop in the Go side, and dequeue callbacks when
> they are available, but that's of course sub-optimal.

You could implement a queue using pthreads and access that from Go
code. The dequeing would be synchronous, so the whole operation would
be synchronous from Go's point of view, even though the queue is
filled concurrently.

Gustavo Niemeyer

unread,
Oct 29, 2010, 6:11:39 PM10/29/10
to Florian Weimer, golang-nuts
Hi Florian,

Yeah, that's precisely what I suggested above. It's not a good
solution on itself because it requires polling.

Florian Weimer

unread,
Oct 30, 2010, 2:29:12 AM10/30/10
to Gustavo Niemeyer, golang-nuts
* Gustavo Niemeyer:

> Hi Florian,
>
>>> The thread-safe and recommended way that Zookeeper can be wrapped
>>> spawns a thread in the background to handle incoming events such as
>>> watches on nodes.  Ian was telling me that there's no way for that
>>> thread to callback into Go code, though.  The potential workaround I
>>> can see is to busy loop in the Go side, and dequeue callbacks when
>>> they are available, but that's of course sub-optimal.
>>
>> You could implement a queue using pthreads and access that from Go
>> code.  The dequeing would be synchronous, so the whole operation would
>> be synchronous from Go's point of view, even though the queue is
>> filled concurrently.
>
> Yeah, that's precisely what I suggested above. It's not a good
> solution on itself because it requires polling.

If you use a condition value, no polling is required. The cost is a
context switch, which can hardly be avoided.

unread,
Oct 30, 2010, 3:26:18 AM10/30/10
to golang-nuts
On Oct 30, 12:11 am, Gustavo Niemeyer <gust...@niemeyer.net> wrote:
> Hi Florian,
>
> >> The thread-safe and recommended way that Zookeeper can be wrapped
> >> spawns a thread in the background to handle incoming events such as
> >> watches on nodes.  Ian was telling me that there's no way for that
> >> thread to callback into Go code, though.  The potential workaround I
> >> can see is to busy loop in the Go side, and dequeue callbacks when
> >> they are available, but that's of course sub-optimal.
>
> > You could implement a queue using pthreads and access that from Go
> > code.  The dequeing would be synchronous, so the whole operation would
> > be synchronous from Go's point of view, even though the queue is
> > filled concurrently.
>
> Yeah, that's precisely what I suggested above.  It's not a good
> solution on itself because it requires polling.

You can use the following methods for communicating between a C thread
and a goroutine:

- a pipe

- pthread_cond_signal (an example can be found here:
http://github.com/0xe2-0x9a-0x9b/Go-SDL/tree/master/sdl/audio/)

Gustavo Niemeyer

unread,
Oct 30, 2010, 9:22:59 AM10/30/10
to ⚛, golang-nuts
> You can use the following methods for communicating between a C thread
> and a goroutine:
>
> - a pipe
>
> - pthread_cond_signal (an example can be found here:
> http://github.com/0xe2-0x9a-0x9b/Go-SDL/tree/master/sdl/audio/)

Ahh, interesting. You mean maintaining the condition variable in the
C side, and then having Go loop calling a C function and blocking a
thread waiting on the condition. Yeah, that sounds like a reasonable
solution I didn't think of. Will give that a try, thanks!

Gustavo Niemeyer

unread,
Oct 30, 2010, 9:25:40 AM10/30/10
to Florian Weimer, golang-nuts
> If you use a condition value, no polling is required.  The cost is a
> context switch, which can hardly be avoided.

I see what you mean. I didn't think about blocking a thread on the C
side. The context switch shouldn't be a big deal in this specific
case. I'm giving this a try, thanks for the suggestion.

Reply all
Reply to author
Forward
0 new messages