How to export C variables from Go shared libraries?

[Hiroaki Nakamura]

Hi gophers,

Can I export C variables from Go shared libraries?

I would like to write an Apache or nginx modules in Go.

I would like to export module variables like

  * example_module in http://httpd.apache.org/docs/2.4/developer/modguide.html

  * ngx_http_hello_module in https://github.com/usamadar/ngx_hello_world/blob/master/ngx_http_hello_module.c

If it is not possible to export C variables, is there a workaround for

writing Apache or nginx modules in Go?

Thanks in advance!

--

Hioraki Nakamura )hnak...@gmail.com)

[Ian Lance Taylor]

On Thu, Jul 16, 2015 at 6:46 AM, Hiroaki Nakamura <hnak...@gmail.com> wrote:
>
> Can I export C variables from Go shared libraries?

You can only use the //export comment with functions.

However, you can put variables in the C header. For example, you can
write

// int i;
import "C"


func init() {
C.i = 1
}

Then uses of the shared library can refer to i.

At least with the current implementation: there is a catch: the init
function will only be reliably run after some function in the file is
called, because the initialization runs in a separate thread. So
although your C code can refer to the variable, it may not be
initialized unless you can ensure that some Go function is called.

Ian

[Hank Donnay]

Would something like

    //export libInit

    func() libInit() {}

do the job, or might the compiler do deep magjicks and not call init() on an empty function?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

-hank

[Ian Lance Taylor]

On Thu, Jul 16, 2015 at 2:34 PM, Hank Donnay <hdo...@gmail.com> wrote:
>
> Would something like
>
> //export libInit
> func() libInit() {}
>
> do the job, or might the compiler do deep magjicks and not call init() on an
> empty function?

I'm not sure I understand the suggestion. Is there a system for which
a function named libInit is called automatically when a shared library
is loaded? That is not the case on GNU/Linux.

Ian

[Hank Donnay]

No, just suggesting a "manual" init function for a library that doesn't do anything but trigger's the go package's init() to run.

--

-hank

[Ian Lance Taylor]

On Thu, Jul 16, 2015 at 6:20 PM, Hank Donnay <hdo...@gmail.com> wrote:
> No, just suggesting a "manual" init function for a library that doesn't do
> anything but trigger's the go package's init() to run.

Sure, that will work.

[Hiroaki Nakamura]

Thanks for your advices.

I made a github project and did an experiment on Linux and OSX.

I tested two versions:

* Not calling libInit():  https://github.com/hnakamur/export_c_variable_from_go_dll_experiment/tree/0972d57461b091aed35c9ed083af0edae0c885cc

* Explicitly calling libInit():

 https://github.com/hnakamur/export_c_variable_from_go_dll_experiment/tree/698d0c598799138f3e70bdf3db6e053d07e264ec

## Not calling libInit()

* On OSX, I got a link error:

  ld: 1 duplicate symbol for architecture x86_64

 https://github.com/hnakamur/export_c_variable_from_go_dll_experiment/tree/0972d57461b091aed35c9ed083af0edae0c885cc#os-x
* On Linux

  * I got a segmentation fault when calling dlclose.

  * i is not set to 1, it remains 0.

 https://github.com/hnakamur/export_c_variable_from_go_dll_experiment/tree/0972d57461b091aed35c9ed083af0edae0c885cc#load-a-shared-library-at-runtime-using-dlopen

## Explicitly calling libInit()
i is set to 1 after calling libInit().

https://github.com/hnakamur/export_c_variable_from_go_dll_experiment/tree/698d0c598799138f3e70bdf3db6e053d07e264ec#load-a-shared-library-at-runtime-using-dlopen

I have two questions:

* How can I a fix a segmentation fault while calling dlclose()?

* Is it possible to running init() without calling libInit()?

Thanks!

--

Hioraki Nakamura )hnak...@gmail.com)

[Fredrik Ehnbom]

>  init() without calling libInit()?

init will run without calling libInit; try for example adding usleep(1000*1000) before printing the variable the first time. Calling the dummy libInit function (or any other go function) just guarantees that all init functions have finished.

[Hiroaki Nakamura]

Hi Fredrik,

Thanks for your comment.

I tried an example, dlopen, usleep(1000*1000), then referencing i:
https://github.com/hnakamur/export_c_variable_from_go_dll_experiment/tree/1951dfb411dcff8986687ca98f801505397fc6cf

The result summary is:

* dlopen example on Linux
  * i is set to 1 after dlopen then sleep.

  * dlclose now runs successfully without segmentation fault.

* compile time link example on Linux

  * i remains 0

* on osx

   * link error: ld: 1 duplicate symbol for architecture x86_64

I have another question:
How can I know the go initialization thread finishes from C?

I welcome further comments.

Thanks!

--

Hioraki Nakamura )hnak...@gmail.com)

[Ian Lance Taylor]

On Fri, Jul 17, 2015 at 9:47 AM, Hiroaki Nakamura <hnak...@gmail.com> wrote:
>
> * dlclose now runs successfully without segmentation fault.

I would not count on dlclose working.

> How can I know the go initialization thread finishes from C?

Currently you can't. Calling a Go function waits for the
initialization thread to complete, but there is no way to wait without
calling a Go function. Please open an issue for this. Thanks.

Ian

[Hiroaki Nakamura]

2015-07-18 7:49 GMT+09:00 Ian Lance Taylor <ia...@golang.org>:

On Fri, Jul 17, 2015 at 9:47 AM, Hiroaki Nakamura <hnak...@gmail.com> wrote:
>
>   * dlclose now runs successfully without segmentation fault.

I would not count on dlclose working.

Could you tell me why, please?

I thought dlclose is working because "after dlclose" is printed

with code at https://github.com/hnakamur/export_c_variable_from_go_dll_experiment/blob/1951dfb411dcff8986687ca98f801505397fc6cf/runtime_load.c#L18

 

> How can I know the go initialization thread finishes from C?

Currently you can't.  Calling a Go function waits for the
initialization thread to complete, but there is no way to wait without
calling a Go function.  Please open an issue for this.  Thanks.

Alright. I opened an issue at https://github.com/golang/go/issues/11768

Now what to do about these two problems?

* compile time link example on Linux

  * i remains 0

* on osx

   * link error: ld: 1 duplicate symbol for architecture x86_64

Thanks!

--

Hioraki Nakamura )hnak...@gmail.com)

[Ian Lance Taylor]

On Fri, Jul 17, 2015 at 4:55 PM, Hiroaki Nakamura <hnak...@gmail.com> wrote:
> 2015-07-18 7:49 GMT+09:00 Ian Lance Taylor <ia...@golang.org>:
>>
>> On Fri, Jul 17, 2015 at 9:47 AM, Hiroaki Nakamura <hnak...@gmail.com>
>> wrote:
>> >
>> > * dlclose now runs successfully without segmentation fault.
>>
>> I would not count on dlclose working.
>
> Could you tell me why, please?
> I thought dlclose is working because "after dlclose" is printed
> with code at
> https://github.com/hnakamur/export_c_variable_from_go_dll_experiment/blob/1951dfb411dcff8986687ca98f801505397fc6cf/runtime_load.c#L18

The Go runtime starts threads on which it runs goroutines. If any of
those threads are running when you call dlclose, your program is
likely to crash. Currently there is no way to cleanly shut down the
Go runtime.

> Now what to do about these two problems?
>
> * compile time link example on Linux
> * i remains 0

This is the problem we have been talking about, isn't it?

> * on osx
> * link error: ld: 1 duplicate symbol for architecture x86_64

I don't know. Is there an issue for this one?

Ian

[Hiroaki Nakamura]

2015-07-18 11:40 GMT+09:00 Ian Lance Taylor <ia...@golang.org>:

On Fri, Jul 17, 2015 at 4:55 PM, Hiroaki Nakamura <hnak...@gmail.com> wrote:
> 2015-07-18 7:49 GMT+09:00 Ian Lance Taylor <ia...@golang.org>:
>>
>> On Fri, Jul 17, 2015 at 9:47 AM, Hiroaki Nakamura <hnak...@gmail.com>
>> wrote:
>> >
>> >   * dlclose now runs successfully without segmentation fault.
>>
>> I would not count on dlclose working.
>
> Could you tell me why, please?
> I thought dlclose is working because "after dlclose" is printed
> with code at
> https://github.com/hnakamur/export_c_variable_from_go_dll_experiment/blob/1951dfb411dcff8986687ca98f801505397fc6cf/runtime_load.c#L18

The Go runtime starts threads on which it runs goroutines.  If any of
those threads are running when you call dlclose, your program is
likely to crash.  Currently there is no way to cleanly shut down the
Go runtime.

I see. Thanks for your expression.

 

> Now what to do about these two problems?
>
> * compile time link example on Linux
>   * i remains 0

This is the problem we have been talking about, isn't it?

No, it is a different problem.

https://github.com/hnakamur/export_c_variable_from_go_dll_experiment/blob/master/compiletime_load.c

```

#include <stdio.h>

#include <dlfcn.h>

#include "libgodll.h"

int main(int argc, char **argv) {

printf("compiletime_load started\n");

usleep(1000 * 1000);

printf("i=%d\n", i);

return 0;

}

```

The code above prints `i=0` instead of `i=1`.

Please see
https://github.com/hnakamur/export_c_variable_from_go_dll_experiment#link-a-shared-library-at-compile-time
for the whole output.

> * on osx
>    * link error: ld: 1 duplicate symbol for architecture x86_64

I don't know.  Is there an issue for this one?

I think there is none.
I searched issues for "is:issue is:open duplicate symbol"
https://github.com/golang/go/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+duplicate+symbol

--

Hioraki Nakamura )hnak...@gmail.com)

[Ian Lance Taylor]

On Fri, Jul 17, 2015 at 9:09 PM, Hiroaki Nakamura <hnak...@gmail.com> wrote:
>
> No, it is a different problem.
>
> https://github.com/hnakamur/export_c_variable_from_go_dll_experiment/blob/master/compiletime_load.c
>
> ```
> #include <stdio.h>
> #include <dlfcn.h>
> #include "libgodll.h"
>
> int main(int argc, char **argv) {
> printf("compiletime_load started\n");
> usleep(1000 * 1000);
> printf("i=%d\n", i);
> return 0;
> }
> ```
>
> The code above prints `i=0` instead of `i=1`.

Yes. You need to call a Go function in order for initialization
functions to run. Calling sleep is not reliable.

>> > * on osx
>> > * link error: ld: 1 duplicate symbol for architecture x86_64
>>
>> I don't know. Is there an issue for this one?
>
>
> I think there is none.
> I searched issues for "is:issue is:open duplicate symbol"
> https://github.com/golang/go/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+duplicate+symbol

I don't think any of those issues are for the same thing.

It may be that GCC and clang have different defaults for whether "int
i;" is a common symbol. It may be necessary to use something like

#cgo CFLAGS "-fcommon"

assuming that works with clang.

Ian

[Hiroaki Nakamura]

2015-07-18 14:37 GMT+09:00 Ian Lance Taylor <ia...@golang.org>:

On Fri, Jul 17, 2015 at 9:09 PM, Hiroaki Nakamura <hnak...@gmail.com> wrote:
>
> No, it is a different problem.
>
> https://github.com/hnakamur/export_c_variable_from_go_dll_experiment/blob/master/compiletime_load.c
>
> ```
> #include <stdio.h>
> #include <dlfcn.h>
> #include "libgodll.h"
>
> int main(int argc, char **argv) {
> printf("compiletime_load started\n");
> usleep(1000 * 1000);
> printf("i=%d\n", i);
> return 0;
> }
> ```
>
> The code above prints `i=0` instead of `i=1`.

Yes.  You need to call a Go function in order for initialization
functions to run.  Calling sleep is not reliable.

I did another experiment with calling the Go function libInit() explicitly.

However i still remains even after calling libInit() followed by usleep().

https://github.com/hnakamur/export_c_variable_from_go_dll_experiment/blob/6cc244e926c109f0f32cd3ed25785925e795809d/compiletime_load.c

```

#include <stdio.h>

#include <dlfcn.h>

#include "libgodll.h"

int main(int argc, char **argv) {

printf("compiletime_load started\n");

printf("#1 i=%d\n", i);

libInit();

printf("#2 i=%d\n", i);

usleep(1000 * 1000);

printf("#3 i=%d\n", i);

return 0;

}

```

The result output on Linux:
  https://github.com/hnakamur/export_c_variable_from_go_dll_experiment#link-a-shared-library-at-compile-time
The result output on OSX:
  https://github.com/hnakamur/export_c_variable_from_go_dll_experiment#link-a-shared-library-at-compile-time-1

>> > * on osx
>> >    * link error: ld: 1 duplicate symbol for architecture x86_64

It may be that GCC and clang have different defaults for whether "int
i;" is a common symbol.  It may be necessary to use something like

#cgo CFLAGS "-fcommon"

assuming that works with clang.

That did the trick! Thanks for your advice.

I added the line below:
//#cgo CFLAGS: "-fcommon"
https://github.com/hnakamur/export_c_variable_from_go_dll_experiment/commit/c5092cea2dc7b4ae5009392ec8b1aa7529bdf166#diff-1

--

Hioraki Nakamura )hnak...@gmail.com)

[Ian Lance Taylor]

Ah, yes, you're right. Sorry.

It's because Go shared libraries are created using -Bsymbolic. That
causes the i in the Go library to be different from the i in the main
executable. This won't matter if you use dlopen and dlsym to access
the variable. But it will matter if you use a simple "extern int i;"
and link against the shared library directly.

Unfortunately, at present, I don't think there is any reliable way for
a Go shared library to export a variable to C code.

Ian

[Hiroaki Nakamura]

OK, now I am clear about the current situation.

Thank you very much for your assistance!

Best regards, 

Hioraki Nakamura