buildmode=c-archive and statically linking into another lib

[Justin Israel]

I'm wondering if someone can tell me if this is either a limitation of buildmode=c-archive static libs, or if I am really just doing something wrong.

Problematic steps:

1. export functions to  libgofoo.a using buildmode=c-archive
   
2. statically link libgofoo.a into another library to produce libfoo.a
3. Someone else consumes libfoo.a and libgofoo.a in their library, and get linker errors about:
   ld: libgofoo.a(go.o): relocation R_X86_64_TPOFF32 against `runtime.tlsg` can not be used when making a shared object; recompile with -fPIC
   libgofoo.a: could not read symbols: Bad value
   

But library that wants to consume these libs can do so if they dynamically link with shared libs from c-shared. Also, if the consumer of the libs is a main binary, then the static linking works and does not complain. It is only when the target is a library.

I could swear I went through and ensured -fPIC was being used, but it didn't seem to make a difference. Am I infact just messing up my compilation settings and failing to add this flag as it suggests, or is there a known limitation related to the steps I have outlined?

Thanks!

Justin

[Ian Lance Taylor]

It looks like the TPOFF32 relocation is coming from the Go code. You
should try building your Go code with the Go equivalent of -fPIC,
which is `go build -buildmode=c-archive -gcflags=-shared`.

Ian

[Justin Israel]

I gave this a try, and I see the "-shared" flag showing up in the "compile" command. But I still get the same error. Is it a factor of the Go installation? Does something in the stdlib need to be built a certain way, since it refers to "go.o" as the problem?

 

Ian

[Ian Lance Taylor]

That's true, you may need to force the standard library to be rebuilt.
The easiest way would probably be something like
go build -buildmode=c-archive -gcflags=-shared -installsuffix _shared -a

Ian

[Justin Israel]

Just tried that as well. I do see it doing the added work, building the stdlib and using the _shared suffix, etc. But when it goes to link the C++ lib it still complains about "(go.o) relocation R_X86_64_TPOFF32 ... ; recompile with -fPIC"

I do appreciate your suggestions so far. I'm surprised it is still giving me the same problem, since what suggested makes perfect sense to fix it. 

 

Ian

[Ian Lance Taylor]

On Wed, May 18, 2016 at 4:47 PM, Justin Israel <justin...@gmail.com> wrote:
>
> Just tried that as well. I do see it doing the added work, building the
> stdlib and using the _shared suffix, etc. But when it goes to link the C++
> lib it still complains about "(go.o) relocation R_X86_64_TPOFF32 ... ;
> recompile with -fPIC"

Sorry, at this point I don't know where it is coming from.

Ian

[Justin Israel]

Its ok. Thanks for trying though. I get by right now by only statically linking if its an executable. Otherwise people have to use dynamic linking.

 

Ian

[Michael Hudson-Doyle]

You'll need -asmflags -shared as well, perhaps?

> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Justin Israel]

On Thu, May 19, 2016 at 12:16 PM Michael Hudson-Doyle <michael...@canonical.com> wrote:

You'll need -asmflags -shared as well, perhaps?

That was it!

Final added flags to "go build":

-gcflags=-shared -asmflags=-shared -installsuffix=_shared -a 

Thanks so much, Ian and Michael!

[Justin Israel]

I've got one more question, related to this process, now that this library is seeing some production use...

The C++ Go binding shared library is being used as a dependency in another library. This other library is a plugin that gets loaded via dlopen. I am seeing the following:

dlopen: cannot load any more object with static TLS

In doing some research, I found that this happens when shared libs default or use "-ftls-model=initial-exec" and/or don't use "-fpic". So it a situation of the big pool of plugins that get dlopen'd into this particular application, and the static TLS slots running out when enough of them use initial-exec. 

I was looking at my shared lib that is created via "go build -buildmode=c-shared -gcflags=-shared -asmflags=-shared -installsuffix=_shared -a" and I do see it using "-fPIC". After doing some SO research, someone suggested doing the following to determine the TLS mode that is being used:

http://stackoverflow.com/questions/22983986/is-there-a-way-to-determine-thread-local-storage-model-used-by-a-library-on-linu

$ readelf -d libgofileseq.so | grep FLAGS

... (FLAGS)    SYMBOLIC STATIC_TLS

It seems no combination of trying to add "-ftls-model=global-dynamic" to my compile process can resolve the problem. Are there any suggestions of where this change can be applied, to correct the situation where dlopen fails? We have found that if this plugin using this shared library gets loaded first, then we don't encounter the static TLS failure. 

Justin

[Michael Hudson-Doyle]

On 25 May 2016 at 11:14, Justin Israel <justin...@gmail.com> wrote:
> I've got one more question, related to this process, now that this library
> is seeing some production use...
>
> The C++ Go binding shared library is being used as a dependency in another
> library. This other library is a plugin that gets loaded via dlopen. I am
> seeing the following:
>
> dlopen: cannot load any more object with static TLS
>
> In doing some research, I found that this happens when shared libs default
> or use "-ftls-model=initial-exec" and/or don't use "-fpic". So it a
> situation of the big pool of plugins that get dlopen'd into this particular
> application, and the static TLS slots running out when enough of them use
> initial-exec.

Welcome to the world of TLS arcana :-)

> I was looking at my shared lib that is created via "go build
> -buildmode=c-shared -gcflags=-shared -asmflags=-shared
> -installsuffix=_shared -a" and I do see it using "-fPIC". After doing some
> SO research, someone suggested doing the following to determine the TLS mode
> that is being used:
>
> http://stackoverflow.com/questions/22983986/is-there-a-way-to-determine-thread-local-storage-model-used-by-a-library-on-linu
>
> $ readelf -d libgofileseq.so | grep FLAGS
> ... (FLAGS) SYMBOLIC STATIC_TLS
>
> It seems no combination of trying to add "-ftls-model=global-dynamic" to my
> compile process can resolve the problem. Are there any suggestions of where
> this change can be applied, to correct the situation where dlopen fails?

No.

The problem is that the Go compiler generates code using the initial
exec model. You might think that it would be possible to change the
compiler to use the global dynamic model, but that's not really
possible because the global dynamic model involves calling a function
from libc for each access. Not only would this be bad for performance
as g is accessed in the prologue of every function, but also this
function at least in some circumstances can use large amounts of
stack. The "tls desc" model described in
http://www.fsfla.org/~lxoliva/writeups/TLS/RFC-TLSDESC-x86.txt would
be more suitable, but you'd need to reimplement _dl_tlsdesc_dynamic in
the Go runtime to handle the 'hard' case on the systemstack.

A more realistic change would be to change the Go runtime on amd64 to
dedicate a register to holding g, as is done on most other ports.

I guess you could also patch your libc to increase TLS_STATIC_SURPLUS...

> We
> have found that if this plugin using this shared library gets loaded first,
> then we don't encounter the static TLS failure.

Are you saying that if you load a bunch of plugins that do not have
STATIC_TLS set, then load one that does, you get an error? Whereas if
you load the STATIC_TLS one first, all is OK? That's a bit surprising
to me, but I don't know all the details for sure.

Cheers,
mwh

> Justin
>
>
>
> On Wednesday, May 18, 2016 at 3:11:57 PM UTC+12, Justin Israel wrote:
>>
>> I'm wondering if someone can tell me if this is either a limitation of
>> buildmode=c-archive static libs, or if I am really just doing something
>> wrong.
>>
>> Problematic steps:
>>
>> export functions to libgofoo.a using buildmode=c-archive
>> statically link libgofoo.a into another library to produce libfoo.a
>> Someone else consumes libfoo.a and libgofoo.a in their library, and get
>> linker errors about:
>> ld: libgofoo.a(go.o): relocation R_X86_64_TPOFF32 against `runtime.tlsg`
>> can not be used when making a shared object; recompile with -fPIC
>> libgofoo.a: could not read symbols: Bad value
>>
>> But library that wants to consume these libs can do so if they dynamically
>> link with shared libs from c-shared. Also, if the consumer of the libs is a
>> main binary, then the static linking works and does not complain. It is only
>> when the target is a library.
>>
>> I could swear I went through and ensured -fPIC was being used, but it
>> didn't seem to make a difference. Am I infact just messing up my compilation
>> settings and failing to add this flag as it suggests, or is there a known
>> limitation related to the steps I have outlined?
>>
>> Thanks!
>> Justin
>>

[Justin Israel]

This is some great information. Thanks for explaining!

I do wish the situation was as you said, for the amd64 port. It would be a bit of a big deal to patch our glibc, and I am told that glibc received some patches surrounding static and dynamic TLS in a newer linux version that we are still testing and have yet to deploy. 

 

> We
> have found that if this plugin using this shared library gets loaded first,
> then we don't encounter the static TLS failure.

Are you saying that if you load a bunch of plugins that do not have
STATIC_TLS set, then load one that does, you get an error? Whereas if
you load the STATIC_TLS one first, all is OK? That's a bit surprising
to me, but I don't know all the details for sure.

Yes, we had the same problem with libgomp. If it was loaded early, it managed to avoid the STATIC_TLS errors. But when loaded later it would have problems. 

[Chao Chen]

Do you any progress to solve the problem? I meet the same problem.

when I call golang build shared library on python from alpine 3.9 container.

failed with OSError: Error relocating ./cc.so: : initial-exec TLS resolves to dynamic definition in ./cc.so

/usr/src/app # go build -o cc.so -buildmode=c-shared main.go

/usr/src/app # readelf -d cc.so

Dynamic section at offset 0x10cd10 contains 22 entries:

Tag Type Name/Value

0x0000000000000001 (NEEDED) Shared library: [libc.musl-x86_64.so.1]

0x0000000000000010 (SYMBOLIC) 0x0

0x000000000000000c (INIT) 0x42000

0x000000000000000d (FINI) 0x92ed9

0x0000000000000019 (INIT_ARRAY) 0xa2078

0x000000000000001b (INIT_ARRAYSZ) 8 (bytes)

0x000000006ffffef5 (GNU_HASH) 0x270

0x0000000000000005 (STRTAB) 0xa50

0x0000000000000006 (SYMTAB) 0x378

0x000000000000000a (STRSZ) 1026 (bytes)

0x000000000000000b (SYMENT) 24 (bytes)

0x0000000000000003 (PLTGOT) 0x10deb0

0x0000000000000002 (PLTRELSZ) 720 (bytes)

0x0000000000000014 (PLTREL) RELA

0x0000000000000017 (JMPREL) 0x41a00

0x0000000000000007 (RELA) 0xe58

0x0000000000000008 (RELASZ) 265128 (bytes)

0x0000000000000009 (RELAENT) 24 (bytes)

0x000000000000001e (FLAGS) SYMBOLIC BIND_NOW STATIC_TLS

0x000000006ffffffb (FLAGS_1) Flags: NOW NODELETE

0x000000006ffffff9 (RELACOUNT) 11040

0x0000000000000000 (NULL) 0x0

/usr/src/app # python test.py

Traceback (most recent call last):

File "test.py", line 2, in 

lib = ctypes.cdll.LoadLibrary('./cc.so')

File "/usr/lib/python2.7/ctypes/init.py", line 444, in LoadLibrary

return self._dlltype(name)

File "/usr/lib/python2.7/ctypes/init.py", line 366, in init

self._handle = _dlopen(self._name, mode)

OSError: Error relocating ./cc.so: : initial-exec TLS resolves to dynamic definition in ./cc.so

[Justin Israel]

On Friday, April 12, 2019 at 1:21:25 PM UTC+12, Chao Chen wrote:
> Do you any progress to solve the problem? I meet the same problem.
>
>

> when I call golang build shared library on python from alpine 3.9 container.failed with OSError: Error relocating ./cc.so: : initial-exec TLS resolves to dynamic definition in ./cc.so

It looks like there was some recent activity that relates to both your musl compiler issue and my original issue:
https://github.com/golang/go/issues/29674

This is specifically targeting Android issues it seems.

I stopped trying to use Go C shared libraries in C++ libs/plugins after this post, since it couldn't cope with large shared plugin environment platforms where there was a race to a limited amount of static TLS slots. But now this is becoming relevant for me again with the new work being done in gopy to generate python extensions:
https://github.com/go-python/gopy/pull/180

> On Wednesday, May 25, 2016 at 7:14:02 AM UTC+8, Justin Israel wrote:
> I've got one more question, related to this process, now that this library is seeing some production use...
>
>
> The C++ Go binding shared library is being used as a dependency in another library. This other library is a plugin that gets loaded via dlopen. I am seeing the following:
>
>
> dlopen: cannot load any more object with static TLS
>
>
> In doing some research, I found that this happens when shared libs default or use "-ftls-model=initial-exec" and/or don't use "-fpic". So it a situation of the big pool of plugins that get dlopen'd into this particular application, and the static TLS slots running out when enough of them use initial-exec. 
>
>
> I was looking at my shared lib that is created via "go build -buildmode=c-shared -gcflags=-shared -asmflags=-shared -installsuffix=_shared -a" and I do see it using "-fPIC". After doing some SO research, someone suggested doing the following to determine the TLS mode that is being used:
>
>
> http://stackoverflow.com/questions/22983986/is-there-a-way-to-determine-thread-local-storage-model-used-by-a-library-on-linu
>
>
>
> $ readelf -d libgofileseq.so | grep FLAGS
> ... (FLAGS)    SYMBOLIC STATIC_TLS
>
>
>
> It seems no combination of trying to add "-ftls-model=global-dynamic" to my compile process can resolve the problem. Are there any suggestions of where this change can be applied, to correct the situation where dlopen fails? We have found that if this plugin using this shared library gets loaded first, then we don't encounter the static TLS failure. 
>
>
> Justin
>
>
>
>
>
> On Wednesday, May 18, 2016 at 3:11:57 PM UTC+12, Justin Israel wrote:
> I'm wondering if someone can tell me if this is either a limitation of buildmode=c-archive static libs, or if I am really just doing something wrong.
>
>
> Problematic steps:
> export functions to  libgofoo.a using buildmode=c-archive

> statically link libgofoo.a into another library to produce libfoo.aSomeone else consumes libfoo.a and libgofoo.a in their library, and get linker errors about:

[w1252...@gmail.com]

I find the package you build with 'go build buildmode=c-archive/c-shared', when you build you code with the lib, the outer need to have main func, I don't know why.

在 2016年5月18日星期三 UTC+8上午11:11:57，Justin Israel写道：

[Justin Israel]

On Wed, Aug 28, 2019, 3:47 PM <w1252...@gmail.com> wrote:

I find the package you build with 'go build buildmode=c-archive/c-shared', when you build you code with the lib, the outer need to have main func, I don't know why.

That requirement is documented here:

https://golang.org/cmd/go/#hdr-Build_modes

在 2016年5月18日星期三 UTC+8上午11:11:57，Justin Israel写道：

I'm wondering if someone can tell me if this is either a limitation of buildmode=c-archive static libs, or if I am really just doing something wrong.

Problematic steps:

1. export functions to  libgofoo.a using buildmode=c-archive
   
2. statically link libgofoo.a into another library to produce libfoo.a
3. Someone else consumes libfoo.a and libgofoo.a in their library, and get linker errors about:
   ld: libgofoo.a(go.o): relocation R_X86_64_TPOFF32 against `runtime.tlsg` can not be used when making a shared object; recompile with -fPIC
   libgofoo.a: could not read symbols: Bad value
   

But library that wants to consume these libs can do so if they dynamically link with shared libs from c-shared. Also, if the consumer of the libs is a main binary, then the static linking works and does not complain. It is only when the target is a library.

I could swear I went through and ensured -fPIC was being used, but it didn't seem to make a difference. Am I infact just messing up my compilation settings and failing to add this flag as it suggests, or is there a known limitation related to the steps I have outlined?

Thanks!

Justin

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/EhndTzcPJxQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/2032198a-00dd-40a8-a8aa-489e9ae406bf%40googlegroups.com.

[w1252...@gmail.com]

thank you~I got it.

在 2019年8月28日星期三 UTC+8上午11:59:42，Justin Israel写道：

To unsubscribe from this group and all its topics, send an email to golan...@googlegroups.com.