Can (go) callbacks created with syscall.NewCallback be called (back) from a threaded DLL in Windows ?

[vanilla blue]

 I am creating callbacks and registering them to be called by some threads (_beginthread) in a DLL. I use LoadLibrary, GetProcAddess and NewCallback for the purpose. 

Expected behavior is that the thread can then proceed to "call into" the callback (through a function pointer which has been generated by syscall.NewCallback).. this taking advantage of all the asm handing finess that is runtime.callbackasm .. 

The problem i am having is that the thread never ends up in the go callback.. it doesnt SIGSEGV either.. I cannot step through the dll as gdb wont cooperate.. thought id ping the group to find out if this is even possible ?

I read that cgo supports callbacks into go from foreign C threads.. 

How does this map to syscall.callbacks into dlls (which create threads) ? I assumed that since __stdcall style callbacks are supported it should be possible.. But since i am no expert on the interplay between _beginthread magic and go callback setup, i defer to experts on the group.. 

Could anyone shed some light on this ? Is my expectation or method of interfacing right ?

Best Regards

Sven

[brainman]

I am not gdb wizzard myself. When everything else fails, I just compile 'INT $3' in my asm and gdb will never fail to stop on that. It is like using println in Go - simple but effective. You would need to recompile runtime package every time you change it by 'go install runtime'. Put 'INT $3' somewhere where you can see it break for sure first, then, once it works, move it into places where you want to debug - somewhere in the start of callback (runtime.callbackasm1). You should see your program stop there, otherwise you dll does not call your callback (on another thread or not).

Once that works, make sure dll call into callback is stdcall, because syscall.NewCallback will assume that.

As to the new thread, everything I have seen calls callback on the same thread. But since syscall.NewCallback uses common runtime.cgocallback_gofunc procedure, it will create new 'm' for your callback if it is started on new thread. This might be all you need. Perhaps there are other show stoppers here, but I can't think of them now. You would have to try and see.

What are you trying to do? Perhaps there are simpler ways of doing it.

Alex

[sven johansson]

Hi Alex,

My last post went missing.. Thanks for the help! Here is a summary of what i am trying to do.. 

I am trying to create a golang interface to a legacy multi-threaded dll which communicates via a callback mechanism for a client. I load the dll register a callback with it. the DLL controller then spawns of "child" threads each of which carry out intensive computation (ffts and the like) and then call the registered callback via the pointer. 

Now I used the syscall.Newcallback to register a go function as callback .. This works as long as it is not called from the "child" threads. So this is a limitation in syscall. As i was debugging the code in runtime i realized that runtime.cgocallback_gofunc  was hanging on needm -> lockextra() .. This was because there was no extra "m" available as it assumed. To fix this i had to import "cgo" which fixed that issue. 

But now i am stuck at : 

// Switch stack and make the call.

    MOVQ    DI, SP

    CALL    runtime·cgocallbackg(SB)

Will debug this further.. But the bottom line is that syscall.Newcallback doesnt work if called from threads (those created with _beginthread.).. 

Perhaps a better solution would be to fix the dll to communicate using  a message queue (zeromq) or have a blocking function which go can call to signal events (like some examples have done before).. but am loathe to leave the confines of emacs/liteIde for Visual studio :) But anyway the syscall documentation could have this disclaimer on the limitation of using the callbacks. 

P.S: the callback uses __stdcall so that is not an issue.. 

/Sven

[brainman]

I am on the road at the moment - I don't have computer with me. It is difficult for me to suggest anything - I have to try and see it for myself. If you have an example program to demonstrate (including dll source), please file an issue https://code.google.com/p/go/issues/list, and I will investigate it when I am back next week. No promise of a solution, but I will try. Meantime, maybe you can implement callback in C or something - you can even use some windows sync facilities to communicate that information into Go if you have to.

Alex

[sven johansson]

Hi Alex,

Thanks for looking into this even when ur travelling! Some good news.. Havent been able to replicate the second hanging issue(at switch stack) i had.. Looks like it was a transient problem, perhaps a gdb state issue.... 

The callbacks are now working for me after i did an import "C" before using them.. Before i did the import it was hanging at lockextra () as mentioned before waiting for an extra "m".. this is because the assumption that there is an extra "m" created doesnt hold true.. I rememeber that the needm function comments stated that if cgo is in use then the list is seeded with a single m..  I guess you would understand this behavior better.. So in the end some sort of initialization is needed if you want to use syscall.Newcallbacks (from threads) as i do.. this is probably because you are using runtime.cgocallback_gofunc internally.. 

Do you still want an issue to be raised ? It would be good to either change the behavior of needm (expecting that an extra m is always available) or explicitly state that syscall Newcallbacks has a dependency to cgo init.. 

I will create a simple dll (with source) which spawns a thread that calls a gocallback sometime later.. Perhaps you can find it useful for internal testing.

/Sven

[Eric Gu]

I'm hitting the same problem when using google-webrtc dll. Do you know how to solve it now?

Best Regards

Eric

[brainman]

I think your description is spot on, but with asm I never trust myself until I see it works. Please, create an issue with some test, so we can keep ourselves honest. Since you went that far in your investigation, you might as well try and suggest a solution there.

Alex

[brainman]

I don't know what your problem is. If you can demonstrate the problem, create an issue. Perhaps it is something that can be fixed.

Alex

[sven johansson]

Hi Alex,

Will do so soon once i can create a simple test dll to go with.. 

Eric: Just do an import "C" in your package even though u dont use cgo.. That fixes the initialization problem for me.. Now the callbacks work fine for me.. Let us know if this helps. 

/Sven

[brainman]

Sorry for confusion. I was replying to Eric.

Alex

[brainman]

I have created an issue: https://code.google.com/p/go/issues/detail?id=6751. Lets take this conversation there. Please, let me know if my test is not good, or if you have better suggestions. Also, did you find a solution yet?

Alex

[jh...@mantech.co.kr]

Dear sven

I got a same problem....

Do you solve this problem?

Thanks.

2013년 11월 5일 화요일 오전 6시 49분 40초 UTC+9, sven johansson 님의 말: