Does Go use the same thread when C code calls Go via callbacks?

[Maxi]

Hi,
I have a multithreaded c webserver library which loads more c
libraries as plugins with dlopen(). Is it possible to have a callback
from one of those plugin libraries return to go code and have it
execute in the same c thread?

The goal is to effectively write Go code inside a C plugin. Since Go
doesn't support embedding into C, I'm trying to take the opposite
approach and compile the webserver as a library and embed it into Go.

I have found this example which his pretty close to what I need:
http://golang.org/misc/cgo/testso/

But it's important to me how it actually works...
If I get a callback from one of the c threads into Go, will it all
work as expected and run in the same thread as the C code? I need to
call back into the C code from the same thread again.

[Ian Lance Taylor]

Maxi <trust...@googlemail.com> writes:

> I have a multithreaded c webserver library which loads more c
> libraries as plugins with dlopen(). Is it possible to have a callback
> from one of those plugin libraries return to go code and have it
> execute in the same c thread?

Yes. To ensure you are in the same C thread you should use
runtime.LockOSThread, although at present I think you always will be
anyhow.

> I have found this example which his pretty close to what I need:
> http://golang.org/misc/cgo/testso/
>
> But it's important to me how it actually works...
> If I get a callback from one of the c threads into Go, will it all
> work as expected and run in the same thread as the C code? I need to
> call back into the C code from the same thread again.

Yes.

Ian

[Dmitry Vyukov]

It's disallowed to call into Go from arbitrary C threads.

You can only call into Go from C if C in turn was called from Go.

That is, call from Go into C, then C can call back into Go on the same thread.

[Maxi]

Dimitry, that is good to know, but can you explain what I need to make
it work anyway? I don't know when the c library calls its plugins
which means that I can't do a callback from Go into C at the right
time.... Unless I simply do one at the creation of every webserver
thread, which would somehow "register" the c thread with Go? I
concluded that there has to be some kind of registration from your
post. Because then I had a call from Go into C, and I can arbitrarily
call from that c thread into Go, correct?

I'll try to make a small example work before I do the whole server,
which is quite an undertaking but any help is appreciated since I just
started looking into Go.

[Dmitry Vyukov]

On Wed, May 23, 2012 at 10:56 AM, Maxi <trust...@googlemail.com> wrote:

Dimitry, that is good to know, but can you explain what I need to make
it work anyway? I don't know when the c library calls its plugins
which means that I can't do a callback from Go into C at the right
time....

Yes, so you C code must be executed in Go threads from start.

 

Unless I simply do one at the creation of every webserver
thread, which would somehow "register" the c thread with Go?

There is no "registration of C threads with Go".

 

I
concluded that there has to be some kind of registration from your
post. Because then I had a call from Go into C, and I can arbitrarily
call from that c thread into Go, correct?

Yes, that's correct.

 

 I'll try to make a small example work before I do the whole server,
which is quite an undertaking but any help is appreciated since I just
started looking into Go.

I don't know, perhaps somebody already done this and has better ideas. But what I have in mind is as follows (just an idea, I never tried it).

// Go part

func main() {

  server := C.CreateServer()  // Create object in C land.

  var wg sync.WaitGroup

  wg.Add(N+1)

  go func() {

    C.MainThread(server)  // Main thread in C land (never returns).
    wg.Done()
  }

  for i := 0; i < N; i++ {

    go func() {

      C.WorkerThread(server)  // Add some worker threads to C land (never returns)

      wg.Done()
    }
  }

  wg.Wait()  // Wait for all threads to finish.

}

This way you have a bunch of threads in C land, and since these threads are Go threads, you can call back into Go from them into anytime.

[Maxi]

Hm it doesn't make sense to me that wrapping a c thread in a goroutine
would somehow change the c thread to be able to call back into Go. You
said yourself that there is no registration of C threads in Go, but
your own example suggests that there has to be something that changes
simply by creating the thread from Go.
Apart from that...technically if I do a pthread_create in the
C.WorkerThread() function, I'm not actually running in the created
thread, the main thread is creating those new threads. So technically
I didn't call the c thread that I want a callback from from Go at all.
I guess I don't see the logic in this.

[Dmitry Vyukov]

On Wed, May 23, 2012 at 1:56 PM, Maxi <trust...@googlemail.com> wrote:

Hm it doesn't make sense to me that wrapping a c thread in a goroutine
would somehow change the c thread to be able to call back into Go.

It does change. The thread is create in Go, and so it can call back into Go.

 

You
said yourself that there is no registration of C threads in Go, but
your own example suggests that there has to be something that changes
simply by creating the thread from Go.

Yes, but it's not a registration of externally created thread (pthread_create) with Go, it's just Go thread.

 

Apart from that...technically if I do a pthread_create in the
C.WorkerThread() function, I'm not actually running in the created

I afraid that won't work. Do not create additional threads with pthread_create, use these threads:

      C.WorkerThread(server)  // Add some worker threads to C land (never returns)

they can call back into Go.

[Maxi]

> I afraid that won't work. Do not create additional threads with
> pthread_create, use these threads: C.WorkerThread(server)

Ohhh. So what you mean is that I need to use goroutines *instead* of
pthreads in the entire webserver, meaning the C.WorkerThread() calls
the method that I would normally pass to pthread_create()? Hmm well
that makes more sense now, but wow that would be a major rewrite of
the server, it uses a lot of pthread specific stuff, like
pthread_setspecific() to set thread local variables.. if it's all a Go
routine, then I assume these calls will error out or don't work. Well
maybe it will work because Go internally uses pthreads too, but
relying on that seems like a recipe for disaster.

This makes me wonder what Ian meant when he said that it would all
work like I initially asked - after all, I specifically said "callback
from a c thread into Go", and he said it would work. Currently you
said that it can't work, because the thread has to be a Goroutine, so
I wouldn't call that a c thread anymore..

[Dmitry Vyukov]

On Wed, May 23, 2012 at 2:35 PM, Maxi <trust...@googlemail.com> wrote:

> I afraid that won't work. Do not create additional threads with
> pthread_create, use these threads: C.WorkerThread(server)

Ohhh. So what you mean is that I need to use goroutines *instead* of
pthreads in the entire webserver, meaning the C.WorkerThread() calls
the method that I would normally pass to pthread_create()?

Yup.

 

Hmm well
that makes more sense now, but wow that would be a major rewrite of
the server, it uses a lot of pthread specific stuff, like
pthread_setspecific() to set thread local variables.. if it's all a Go
routine, then I assume these calls will error out or don't work. Well
maybe it will work because Go internally uses pthreads too, but
relying on that seems like a recipe for disaster.

Go uses pthread in cgo mode, so pthread_setspecific() should work.

However, if you just pthread_create here and there, then it's problematic.

 

This makes me wonder what Ian meant when he said that it would all
work like I initially asked - after all, I specifically said "callback
from a c thread into Go", and he said it would work. Currently you
said that it can't work, because the thread has to be a Goroutine, so
I wouldn't call that a c thread anymore..

Well, yes, you may think of it as a goroutine, but it's a goroutine backed up by a real OS thread, so it's a thread as well.

If only want to implement plugins in Go, then the following trick may work.

func main() {

  for i := 0; i < N; i++ {

    go func() {

      for {

        call := C.DequeCall()

        ExecuteCallInGo(call)

        C.NotifyDone(call)

      }

    }()
  }

  C.StartServer()

}

So, C.StartServer() starts your C server, and that function can spawn any other number of threads as it needs.

But all that threads does not directly call back into Go. Instead you have a message queue in C land. When a C thread needs to call back into Go (to execute plugin call), it enqueues a call descriptor into the queue (plugin name, func name, args, etc), and waits for completion notification. Goroutines that call C.DequeCall(), poll that queue; once a message is available, the goroutines returns into Go, executes the call and notifies completion.

This way you separate C land (with it's own threads) and Go land with own threads. Different threads communicate only by means of the message queue (the queue itself must be in C land).

It's not particularly fast way to make callbacks. But if performance is not extremely critical, I think it's the simplest way.

[Ian Lance Taylor]

Maxi <trust...@googlemail.com> writes:

> This makes me wonder what Ian meant when he said that it would all
> work like I initially asked - after all, I specifically said "callback
> from a c thread into Go", and he said it would work. Currently you
> said that it can't work, because the thread has to be a Goroutine, so
> I wouldn't call that a c thread anymore..

A goroutine is not the same as a thread, but when a goroutine calls into
C code it is running in a thread. From the perspective of the C code,
it is a thread. That thread can call back to Go code. Other threads,
those created directly by the C code, currently can not.

It's actually a bug that other threads can not call Go code. There have
been a couple of efforts toward fixing this, but none have completed.

> it uses a lot of pthread specific stuff, like
> pthread_setspecific() to set thread local variables

There is no problem with having a goroutine call C code that uses
pthread_setspecific. The C code will be running in a specific thread.
However, if you want Go code to call C code that calls
pthread_setspecific, and then have the C code return back to the Go
code, and then have the Go code call C code, and have that C code use
pthread_getspecific, then you must use runtime.LockOSThread in the Go
code so that the C code always sees the same thread.

Ian

[Maxi]

I got it working but under load testing there are inexplicable "error:
read() failed: Connection reset by peer (104)" from my server and
about 1/4 of data throughput even without using callbacks. Tracking
such a behavior down to a source is pretty much impossible so I think
this concludes my experiment with Go.

Here is the Go code I used:

var N = 4
func Server(wg sync.WaitGroup) {
runtime.LockOSThread()
fd := C.CreateServer(0, nil)

for i := 0; i < N; i++ {

go Worker(wg)
}
C.StartLoop(fd)
wg.Done()
}

func Worker(wg sync.WaitGroup) {
runtime.LockOSThread()
C.WorkerThread()
wg.Done()
}

func main() {
runtime.LockOSThread()
var wg sync.WaitGroup
wg.Add(N + 1)

go Server(wg)

wg.Wait()

}