multithreaded cgo

[Luke Scott]

I'm using a C++ library, wrapped in C, in Go. The size of the library is very large, and it's impractical to rewrite it in Go (500k+ lines). The library is particular sensitive to changing threads. You have an opening ->Enter() function and a closing ->Exit() function. The calls in-between expect that the thread is the same. I've wrapped each of these calls in C for Go to call (enter, exit, etc...). Because of the sensitivity to threads, I'm currently doing runtime.LockOSThread() before ->Enter(), and runtime.UnlockOSThread() after ->Exit().

Is the called C (cgo) function inside the same thread as the Go routine?

From what I understand there are some ways to call C more directly without cgo. What are they, and what are the implications?

How does any of the above effect GOMAXPROCS, and given that the C++ library is picky about threads, does it matter anyway?

One of the things I have to do with this library is create a data-object from a Go struct/map. In order to do this, I have to do something like this:

runtime.LockOSThread()

C.enter()

obj := C.object_new()

C.object_set(obj, C.CString("key"), C.CString("value")) // probably need a free() for CString()...

// build a nested object...

C.exit()

What would be the optimal way to do this, where potentially hundreds of C calls are being made from Go?

Luke

[Ian Lance Taylor]

On Thu, Aug 29, 2013 at 12:58 PM, Luke Scott <lu...@visionlaunchers.com> wrote:
> I'm using a C++ library, wrapped in C, in Go. The size of the library is
> very large, and it's impractical to rewrite it in Go (500k+ lines). The
> library is particular sensitive to changing threads. You have an opening
> ->Enter() function and a closing ->Exit() function. The calls in-between
> expect that the thread is the same. I've wrapped each of these calls in C
> for Go to call (enter, exit, etc...). Because of the sensitivity to threads,
> I'm currently doing runtime.LockOSThread() before ->Enter(), and
> runtime.UnlockOSThread() after ->Exit().
>
> Is the called C (cgo) function inside the same thread as the Go routine?

Yes.

> From what I understand there are some ways to call C more directly without
> cgo. What are they, and what are the implications?

The other available option is SWIG. It has the same limitations as cgo.

> How does any of the above effect GOMAXPROCS, and given that the C++ library
> is picky about threads, does it matter anyway?

GOMAXPROCS will still limit the number of threads that the Go runtime
will run concurrently, regardless of your calls to LockOSThread. But
the Go runtime will always create at least one unlocked thread as
needed. And non-Go code will not count against the GOMAXPROCS count.

> One of the things I have to do with this library is create a data-object
> from a Go struct/map. In order to do this, I have to do something like this:
>
> runtime.LockOSThread()
> C.enter()
> obj := C.object_new()
> C.object_set(obj, C.CString("key"), C.CString("value")) // probably need a
> free() for CString()...
> // build a nested object...
> C.exit()
>
> What would be the optimal way to do this, where potentially hundreds of C
> calls are being made from Go?

Sorry, I'm not sure what you are asking.

Ian

[Luke Scott]

On Aug 29, 2013, at 1:19 PM, Ian Lance Taylor <ia...@golang.org> wrote:

On Thu, Aug 29, 2013 at 12:58 PM, Luke Scott <lu...@visionlaunchers.com> wrote:

I'm using a C++ library, wrapped in C, in Go. The size of the library is
very large, and it's impractical to rewrite it in Go (500k+ lines). The
library is particular sensitive to changing threads. You have an opening
->Enter() function and a closing ->Exit() function. The calls in-between
expect that the thread is the same. I've wrapped each of these calls in C
for Go to call (enter, exit, etc...). Because of the sensitivity to threads,
I'm currently doing runtime.LockOSThread() before ->Enter(), and
runtime.UnlockOSThread() after ->Exit().

Is the called C (cgo) function inside the same thread as the Go routine?

Yes.

Thanks for clearing that up. For some reason I thought it might be creating a new thread.

From what I understand there are some ways to call C more directly without
cgo. What are they, and what are the implications?

The other available option is SWIG.  It has the same limitations as cgo.

SWIG uses cgo too, doesn't it? So either you use SWIG to build the C wrappers, or you do it yourself? Or am I misunderstanding something about SWIG?

How does any of the above effect GOMAXPROCS, and given that the C++ library
is picky about threads, does it matter anyway?

GOMAXPROCS will still limit the number of threads that the Go runtime
will run concurrently, regardless of your calls to LockOSThread.  But
the Go runtime will always create at least one unlocked thread as
needed.  And non-Go code will not count against the GOMAXPROCS count.

So if a cgo call is blocking it doesn't count because it's considered a syscall?

One of the things I have to do with this library is create a data-object
from a Go struct/map. In order to do this, I have to do something like this:

runtime.LockOSThread()
C.enter()
obj := C.object_new()
C.object_set(obj, C.CString("key"), C.CString("value")) // probably need a
free() for CString()...
// build a nested object...
C.exit()

What would be the optimal way to do this, where potentially hundreds of C
calls are being made from Go?

Sorry, I'm not sure what you are asking.

Having a hard time making sense of it myself. There seems to be multiple ways to call C code. I'm trying to understand what they all are, and what the pros and cons are.

- cgo through C.name

- runtime, like http://golang.org/src/pkg/runtime/string.goc (w/ or w/o .goc extension?)

- ASM style, like http://golang.org/src/pkg/crypto/aes/cipher_asm.go (does this require gccgo?)

I'm essentially going to be calling hundreds of C calls, mixed with some calls to the reflect package in between.

(I'm specifically trying to integrate Go with V8)

Luke

[Luke Scott]

On Aug 29, 2013, at 1:39 PM, Kyle Lemons <kev...@google.com> wrote:

> This, as well as .goc, is actually compiled by the Go c compiler/assembler (6c/6a). It compiles using the go calling conventions and such. Your C++ code is very unlikely to compile under these circumstances.

Is the same true with the gccgo compiler? Should I consider using gcccgo?

Luke

[Ian Lance Taylor]

On Thu, Aug 29, 2013 at 1:33 PM, Luke Scott <lu...@visionlaunchers.com> wrote:
>
> On Aug 29, 2013, at 1:19 PM, Ian Lance Taylor <ia...@golang.org> wrote:
>
>> The other available option is SWIG. It has the same limitations as cgo.
>
>
> SWIG uses cgo too, doesn't it? So either you use SWIG to build the C
> wrappers, or you do it yourself? Or am I misunderstanding something about
> SWIG?

I wouldn't say that SWIG uses cgo, exactly, but SWIG does use the same
mechanism that cgo uses.

> So if a cgo call is blocking it doesn't count because it's considered a
> syscall?

Right.

Ian

[Ian Lance Taylor]

With either gccgo or the gc compiler you need to manage the transition
between Go code and C/C++ code. It's true that when using gccgo you
can directly call code compiled by GCC, but if you do it without care
the Go scheduler will stop. Using cgo or SWIG (both of which support
gccgo) will insert the right transition code. When using gccgo you
can also write that transition code by hand, but I don't see why that
would be a big win for what you are trying to do.

Ian

[Luke Scott]

So every C call from Go is going to be expensive because of the interaction with the scheduler. Since V8 needs the same thread, an entire OS thread is taken up by a single Go routine. And by entering and locking the V8 isolate, I block access by other threads while making these C calls.

I don't suppose there is a way to pass a Go type to C and reflect it from C, is there? I basically need to map Go types to V8 types, but I need to do it as fast as possible. The back-and-forth is probably not good. I need to map nested objects.

Luke

[Ian Lance Taylor]

On Thu, Aug 29, 2013 at 2:38 PM, Luke Scott <lu...@visionlaunchers.com> wrote:
>
> So every C call from Go is going to be expensive because of the interaction with the scheduler. Since V8 needs the same thread, an entire OS thread is taken up by a single Go routine. And by entering and locking the V8 isolate, I block access by other threads while making these C calls.
>
> I don't suppose there is a way to pass a Go type to C and reflect it from C, is there? I basically need to map Go types to V8 types, but I need to do it as fast as possible. The back-and-forth is probably not good. I need to map nested objects.

From your original message it sounds like you need to build the C
objects by using multiple function calls into the library. In that
case I think the most runtime efficient approach is going to be to
write some sort of serialization format. Then you can serialize the
Go code into a []byte, pass the []byte to C, and turn it into the C
object.

Ian

[Kyle Lemons]

.goc is special :).

 

- ASM style, like http://golang.org/src/pkg/crypto/aes/cipher_asm.go (does this require gccgo?)

This, as well as .goc, is actually compiled by the Go c compiler/assembler (6c/6a).  It compiles using the go calling conventions and such.  Your C++ code is very unlikely to compile under these circumstances.

 

I'm essentially going to be calling hundreds of C calls, mixed with some calls to the reflect package in between.

(I'm specifically trying to integrate Go with V8)

Luke

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Dmitry Vyukov]

Agree. I would probably try to use an array of structs instead of []byte.
So first you do all reflect calls, collect data in the array (or
[]byte), then do single cgo call to your C wrapper, then the C wrapper
does multiple calls into V8.

[Gerard]

On Thursday, August 29, 2013 10:33:30 PM UTC+2, Luke Scott wrote:

(I'm specifically trying to integrate Go with V8)

For inspiration take a look at github.com/mattn/go-v8 (github is down now) and at this thread about C++ linking in Go 1.1.2