Proposal: Shared libraries in Go, take 2 (+linux/amd64 implementation)

[Elias Naur]

Hi,

A few months ago, I got some CLs reviewed and submied that added some low level work towards support for creating shared libraries written in Go. The more intrusive runtime changes were never accepted, so I'd like to propose a more clean, simple, and hopefully more acceptable way of supporting shared library in the Go runtime and tools. This proposal uses linux terminology, but shared libraries on the other OSes are (hopefully) similar.

Go shared libraries

Supplying the -shared flag to 6l will output a shared library that is similar to a go program, but with a few notable differences:

Just as a go program includes all the referenced go code and the go runtime, a go shared library is a dynamic shared object (.so file) that includes all the referenced go code and the runtime.

When a go program is executed, the main thread calls into the Go runtime, which is initialized, init functions are run and finally, main() is run. A go library does not own the native program's main thread, and in fact might not even be loaded from the main thread (in the dlopen() case).

Instead, the thread loading the go library spawns a new thread which from then on functions as the Go main thread. The fresh go main thread in turn runs go runtime initialization and then notifies the loading thread to return control to the main program. The runtime initialization for go libraries is the same as for ordinary go programs, except that main() is expected to return, and that go won't exit the program when that happens.

I don't have strong feelings about running main() or not, but running it is the closest to normal go program behaviour, and then main() can double as a "library init" function.

After initialization, the main program can call into the Go library just as if a foreign thread called into Go in a Cgo program.

Deadlock detection is effectively disabled in go libraries, even if no cgo calls have been made from Go, because it can't be known whether the main program is deadlocked or not.

os.Args normally contain the programs arguments. Shared libraries generally don't have access to the program arguments without cooperation from the main program, so os.Args length is 0 for go libraries.

Linux/amd64 implementation

To test the proposal and to make sure it is complete, I've added support for go shared libraries to the linux/amd64 platform. It builds on the existing -shared flag, and on the recent support for external linking and callbacks from foreign threads.

The low level and less intrusive changes are in https://codereview.appspot.com/9733044/. It reimplements -shared in 6l with external linking and adds support for the initial exec TLS model. This CL  contains what I believe is the necessary linker changes for go shared library support, regardless of how the runtime changes and user visible semantics end up to be. If my runtime proposal is not accepted, it is my hope that this CL can be reviewed on any technical problems and submitted anyway.

The runtime changes are in https://codereview.appspot.com/9738047/. It contain the meat of the proposal and I expect this CL to contain the most controversy, if any. The CL also includes a test for a basic C program linking to a go library and a more advanced C program that dlopen()s a go library and use __thread TLS variables.

The implementation has some limits that are not inherent in the proposal:

• Shared library support is not "out of the box", in the sense that Go (on amd64) must be built with the -largemodel flag to gc and cc to support go libraries. Ordinary go programs can still be built with those flags, but the position independent code and the initial exec TLS model makes the resulting executables slightly slower. This restriction is similar to the race detector, which also needs a special build to function. Ideally, the same Go build should support both ordinary go programs and go libraries without any unnecessary speed penalty. Advice on how to implement that in the tooling is welcomed.
• Multiple go libraries loaded in the same native program, or a go program loading a go library probably won't work. An immediate showstopper is that cgo currently exports some symbols (x_cgo_init etc.) in all go libraries and programs. Multiple go libraries are also inefficient, in the sense that they won't share a single runtime, heap, garbage collector, etc.
• No go tool flag ("-shared"?) has been added yet, but can be trivially added.

Linux/arm 

The basic -shared flag support is already present on linux/arm, and I plan to implement this updated proposal for linux/arm too. However, since external linking is now required, I'd like to know if anyone (Russ?) is already working on that? If not, I'd like to go ahead and attempt an implementation of external linking myself.

 - elias

[minux]

i expect the exact semantics of a Go shared library needs some discussion.

On Thu, May 30, 2013 at 4:16 AM, Elias Naur <elias...@gmail.com> wrote:

Linux/arm 

The basic -shared flag support is already present on linux/arm, and I plan to implement this updated proposal for linux/arm too. However, since external linking is now required, I'd like to know if anyone (Russ?) is already working on that? If not, I'd like to go ahead and attempt an implementation of external linking myself.

I'm sure Russ is not working on that. I have plan to implement that, but as i haven't started yet,

feel free to take the work (and please post a note to https://code.google.com/p/go/issues/detail?id=5590

if you've started).

btw, thank you for initiating the shared library work and providing the first working prototype.

[Ian Lance Taylor]

On Wed, May 29, 2013 at 1:16 PM, Elias Naur <elias...@gmail.com> wrote:
>
> Multiple go libraries loaded in the same native program, or a go program
> loading a go library probably won't work. An immediate showstopper is that
> cgo currently exports some symbols (x_cgo_init etc.) in all go libraries and
> programs. Multiple go libraries are also inefficient, in the sense that they
> won't share a single runtime, heap, garbage collector, etc.

This is an interesting approach, but I think it is clear that we need
to understand multiple Go shared libraries without just failing
horribly. Should each shared library live in its own world? That
will not play well with dlopen. Should all the shared libraries live
in the same world? That raises questions about initialization.

Also, if we support dlopen, what should we do for dlclose? E.g., what
if you dlopen a shared library, it calls runtime.SetFinalizer, and
then you dlclose that library?

I think this is good work, but I also think we need to figure out
these questions, at least to the extent that we know where we are
heading even if it doesn't yet work.

Ian

[Elias Naur]

Hi,

Thank you for your response.

(Could someone help me figure out why my two CLs apparently didn't appear on golang-dev even though I hg mail'ed them? I've had this problem before:

https://groups.google.com/forum/?fromgroups#!search/elias$20golang$20nuts/golang-nuts/uiMeN2EjBMI/QqhdjT75BhMJ

But with the proposed changes to how the mailing lists work I'm not sure if CLs are still supposed to automatically appear on golang-dev, especially those from outsiders like myself.

This is important to me, because I (still) think the linker changes from 9733044 are necessary if any Go code is going to end up in a shared library in any form. If so, I'd like (as much as possible from) 9733044 to be considered ready for review or a statement to the effect that the CL must also wait for the eventual discussion of go shared library semantics.)

On Wednesday, May 29, 2013 10:40:31 PM UTC+2, Ian Lance Taylor wrote:

On Wed, May 29, 2013 at 1:16 PM, Elias Naur <elias...@gmail.com> wrote:
>
> Multiple go libraries loaded in the same native program, or a go program
> loading a go library probably won't work. An immediate showstopper is that
> cgo currently exports some symbols (x_cgo_init etc.) in all go libraries and
> programs. Multiple go libraries are also inefficient, in the sense that they
> won't share a single runtime, heap, garbage collector, etc.

This is an interesting approach, but I think it is clear that we need
to understand multiple Go shared libraries without just failing
horribly.  Should each shared library live in its own world?  That
will not play well with dlopen.  Should all the shared libraries live
in the same world?  That raises questions about initialization.

Also, if we support dlopen, what should we do for dlclose?  E.g., what 
if you dlopen a shared library, it calls runtime.SetFinalizer, and 
then you dlclose that library? 

Not being qualified to answer the difficult questions about what to do in the long term for shared libraries in go, I have tried to figure out how to implement simple solutions that works sooner rather than later that also won't get in the way when more general solutions are implemented.

I consider the model where each go shared library lives in its own world in the shared address space, only communicating with each other and the main program through cgo, a fit for that goal. I also consider it acceptable (and again, forward compatible) for dlclose not to be supported/undefined. I also consider failing (less horribly) to load multiple go shared libraries acceptable, because there is a significant overhead for each Go runtime and standard library instance anyway. In fact, I see go library support more as an extension to Cgo, enabling broader interoperability with other programs, a necessary evil in the same way Cgo itself could be considered. 

(Eager to have a answer backed by evidence, I began unexporting all Cgo symbols (except symbols deliberately //export'ed) to demonstrate multiple go shared libraries loaded simultaneously. It was easy to sprinkle __attribute__ ((visibility ("hidden"))) in gcc compiled files, but the cgo pragma flags cgo_export_dynamic and cgo_export_static confused me. The description of the directives in cmd/cgo/doc.go are exactly the same except for a note about compatibility with SWIG. Is SWIG the reason Cgo symbols need to be marked as cgo_export_dynamic, which I assume means exported in the dynamic symbol table, as opposed to only cgo_export_static which I assume means only in the static symbol table? I tried to leave out cgo_export_dynamic from the crosscal2 pragma in pkg/runtime/cgo/callbacks.c, but crosscall2 still appeared in the dynamic symbol table of the resulting shared library.)

 

I think this is good work, but I also think we need to figure out
these questions, at least to the extent that we know where we are
heading even if it doesn't yet work.

May I urge you to consider discussing a simple solution for the shared library use case sooner rather than later (issue 256 is from 2009)? I worry that the much more general and complicated question of how to load Go code at runtime is bogging down and delaying the, to me, somewhat separate issue of interoperability problems where a shared library is the only way to practically communicate with foreign programs. You may recall my use case is running Go code from Android dalvik programs, which is a particularly unfortunate case. It would be quite ironic if darwin/arm on iOS (hi, Minux!) would be supported before dalvik could run Go ;) Judging from the go issue tracker, the bug report votes, and this mailing list I'm sure missing library support is a significant pain point to many others as well.

@Minux: Thanks, I'll start on 5l external linking right away, taking the 5590 lock as I go :)

 - elias

[Ian Lance Taylor]

On Sat, Jun 1, 2013 at 5:38 AM, Elias Naur <elias...@gmail.com> wrote:
>
> May I urge you to consider discussing a simple solution for the shared
> library use case sooner rather than later (issue 256 is from 2009)?

We can have a simple solution as long as we know the ultimate
destination. But my experience is that a simple solution when we
don't know where we are going can lock us into a certain path, a path
that becomes difficult to unwind because people start depending on it.
Sorry to be a wet blanket on this.

One way to avoid that problem is to ensure that the troublesome cases
do not work in any way. For example, ensure that there can be only
one Go shared library in a process. Ensure that a Go library can not
be dlclose'd. That may help flush out use cases.

Ian

[Tad Glines]

I apologize if this question was already answered somewhere else, but...

I'm puzzled as to why you propose putting the runtime in each go shared library. Typically when you have a shared dependency, that code is placed in a separate shared library. On linux at least it's also possible to leave the runtime in the executable if the executable is linked with -rdynamic/--export-dynamic (at least according to the dlopen man page). I'm not aware of a way to do this on Windows so the runtime would have to be in a separate DLL.

Having the runtime in a separate shared library makes it possible to load multiple go shared libraries in the same executable.

So, I guess the question is, was this options considered and discarded and if so, why?

On Wed, May 29, 2013 at 1:16 PM, Elias Naur <elias...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Elias Naur]

I realized that initializing the go runtime from a library initializer is too risky. In particular, starting and waiting for a thread from DllMain will deadlock on windows. So I've moved the runtime initalization to the first cgo callback made by the main program. I considered an explicit InitializeGoRuntime() call exported from all go libraries, but that would defeat the use cases where you cannot add new calls to the main program.

I added the - Bsymbolic linker flag when building go libraries to ensure that any references to global symbols stay within the library. I also added a test for multiple libraries in the same process. The test failed as expected, but not as horribly as I had thought. The failure is a mmap that Go expects to be at a particular address. I used this as inspiration to enable a nicer error message for multiple loaded libraries: When a go shared library is initialized, a page with a constant address is attempted to be mapped. If that fails, Go aborts assuming that another Go library was already loaded.

Unfortunately, I was unable to find a way to block dlclose(). ELF destructors looked promising, but since they also run on process exit I could not discern the dlclose case.

I apologize if this question was already answered somewhere else, but...
I'm puzzled as to why you propose putting the runtime in each go shared library. Typically when you have a shared dependency, that code is placed in a separate shared library. On linux at least it's also possible to leave the runtime in the executable if the executable is linked with -rdynamic/--export-dynamic (at least according to the dlopen man page). I'm not aware of a way to do this on Windows so the runtime would have to be in a separate DLL.

 

Having the runtime in a separate shared library makes it possible to load multiple go shared libraries in the same executable.

 

So, I guess the question is, was this options considered and discarded and if so, why?

I tried to keep it simple, and avoid difficult questions like "what happens if multiple libraries built with different versions of Go share a runtime?" I also tried to keep the changes minimal, so running go from a library is as similar as possible as running a go program.

Comments from rsc in CL 9738047:

All the problems from last year remain: why should the init only happen once but
the main start anew on each call?

Am I misunderstanding something? The go library main.main() only runs once, just after init.

 

Why is it reasonable for os.Args to be empty?

I failed to find a way to reliably access the program arguments from a shared library, so I let that limitation bleed into the design. A way to keeping os.Args would be preferable, but if that is impossible, is that not reasonable enough?

 

What about the environment? And so on.

While handling the missing argv I forgot to handle the environment, sorry. I've added hooks to fetch the environment from the global environ variable from cgo instead. I've also added a test to make sure env keys get through even in library mode.

And so on.

Could you elaborate on what problems from last year still remain and what new problems do you see? In the original runtime CL 6822078 you wrote:

This CL will then contain only the new design decisions about this "library"
mode. This is the part I am fundamentally unsure about. There are complications
having to do with what happens when the m0 exits, how you handle multiple
threads calling into Go simultaneously, why there is no os.Args, whether signal
handlers should persist on return, whether other M's that are started during the
call should be killed at return, and so on. But at least if we get the other
mechanical details done we can have that conversation.

 The design is fundamentally different now, and the go runtime now have its own "main" thread even in library mode, so I'm not sure which of these issues are relevant anymore.

 - elias 

[John Nagle]

On 5/29/2013 1:16 PM, Elias Naur wrote:
> Hi,
>
> A few months ago, I got some CLs reviewed and submied that added some low
> level work towards support for creating shared libraries written in Go.

Shared libraries are possible, but are they desirable?

Shared libraries are a win only for a few use cases. If you
have a server which is running many instances of the same program,
you'll share code (on most sane operating systems). If you have
multiple programs which are completely different, there will be
little code sharing. Only when you have a machine running
multiple programs which are relatively similar and use the
same libraries are shared libraries a win. At best, you
save some code space in RAM.

Shared libraries are a lose in many ways. The program
brings in the entire library, not just the pieces needed.
You're relying on the virtual memory system to throw out
the blocks of unused code. (Paging is on the way out,
anyway; most mobile devices don't use it and if you're
paging on a server, you're overloaded.)

If shared libraries have state in their own shared
data space, they're now really a form of "big object".
Making such things play nicely with Go semantics is hard.
Is it worth it? Should Go go there?

(Microsoft goes overboard on this. Windows has
facilities for "injecting a DLL" into a running
process. Similar facilities have been hacked into Linux.
The primary use case seems to be for malware.)

John Nagle

[Elias Naur]

I think we agree on most of this. I didn't make it clear in the first post, but I consider shared library support as a useful way to extend the reach of cgo, nothing else. In the majority of cases, cgo is enough to interface with native code, but in a few cases, you don't control the main program. Running go code from Android dalvik is a high profile example, but I'm sure there a many others.

My proposed design also reflect this corner case aspect of the feature: I tried to make go libraries behave as close as possible to a go program.

That's why I consider go libraries a net win.

 - elias

 

[GreatOdinsRaven]

One Go core team member's opinion on shared libraries: http://harmful.cat-v.org/software/dynamic-linking/

Go the spec says nothing about libraries. I just don't expect to see such support in the gc implementation. But other implementations? Why not? As long as it doesn't somehow violate the spec, right? Maybe gccgo is a more likely candidate

[Elias Naur]

On Saturday, June 8, 2013 9:22:47 PM UTC+2, GreatOdinsRaven wrote:

One Go core team member's opinion on shared libraries: http://harmful.cat-v.org/software/dynamic-linking/

Go the spec says nothing about libraries. I just don't expect to see such support in the gc implementation. But other implementations? Why not? As long as it doesn't somehow violate the spec, right? Maybe gccgo is a more likely candidate

Again, this proposal is about expanding Go interoperability to cases where the only way to run foreign code is through a shared library. In fact, this proposal specifically does _not_ enable any sharing; *every* go "shared" library is self-contained and will contain all code, including the runtime and the standard library.

 - elias

[John Nagle]

On 6/9/2013 12:40 AM, Elias Naur wrote:

> Again, this proposal is about expanding Go interoperability to cases where
> the only way to run foreign code is through a shared library. In fact, this
> proposal specifically does _not_ enable any sharing; *every* go "shared"
> library is self-contained and will contain all code, including the runtime
> and the standard library.

If if duplicates the whole run-time system and there is no sharing,
you're better off with interprocess communication.

John Nagle

[Elias Naur]

Interprocess communication is fine for many purposes, but for my particular use case (Android game written in Go) the performance penalty and the tediousness of marshalling/unmarshalling calls is too high. IMO, of course. Especially compared to the actual footprint on the runtime of this feature; the linker already supports external linking, and the runtime already supports callbacks from foreign threads. Those two are the heavy lifters in go shared libraries - my proposal simply adds a special initialization mode on top.

 - elias

[Elias Naur]

https://code.google.com/p/go/source/detail?r=1d5a80b07916

- elias

[Elias Naur]

I've added support for linux/arm shared libraries at

https://codereview.appspot.com/9738047/

https://codereview.appspot.com/9733044/

As well as the prerequisite support for external linking on ARM:

https://codereview.appspot.com/10271047/

https://codereview.appspot.com/10499043/

 - elias

[capnm]

Thanks for working on this.

FYI I'd tried the 2 "external linking" CLs on current tip, it seems to be broken:

https://github.com/capnm/golang/issues/1

[Elias Naur]

Thank you for testing this.

runtime·read_tls_fallback is used when GOARM < 7 so I tried "GOARM=5 ./all.bash" to simulate that case, but failed to reproduce that error. It could be a problem in how the CLs was applied. Testing a clean checkout of go, it appears that hg clpatch doesn't deal well with overlapping CLs.

(Since the changes are somewhat extensive and isolated I used the hg mq extension to manage the changes. The codereview extension doesn't work well with mq, so I created the CLs one by one by hg qfinish'ing previous mq patches and then hg change/upload the diffs.)

I've attached the mq patches so you can hg qimport them. You'll need "merge-gm" and "5l-ext-linking" to enable external linking, and "shared-lib" and "shared-lib-runtime" to enable shared library support.

 - elias

[tthatch90]

I'm trying to apply the latest diffs against the go source so I can play around with or work on this, but I'm unable to find a tag or release to which it applies cleanly.

Is there a particular Mercurial tag or release rev you can give me that your latest complete patch will apply cleanly against?

I'm sorry if there is some obvious answer to this I'm missing but going through the docs I can find about how codereview works I'm not finding it how to to cross reference what's there against Mercurial revs.

thanks for taking the time to champion this so to speak, likewise I can really use this feature, we may implement a custom version of go to get it.

[Elias Naur]

It's been so long since I last worked on it, so I don't have a revision that the CL applies cleanly to. My incentive for shared library support was to get Go working in Android apps, and now that Go 1.4 seems to include that, I'm less inclined to further work on this feature. FWIW, I think the official Go 1.4 wishlist includes a design document for shared libraries, so it seems that official support for shared libraries might not be too distant.

 - elias

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/zmjXkGrEx6Q/unsubscribe.

To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Chao Chen]

I don't understand initial exec TLS model. But I meet the following problem when use go shared library. 

When I load Go shared library from python2.7 script on alpine 3.9 container.

Would you please give some suggestions? I am so appreciated for your help!

When I call Go shared library from python2.7 script on alpine 3.9 container.

failed with OSError: Error relocating ./cc.so: : initial-exec TLS resolves to dynamic definition in ./cc.so

/usr/src/app # go build -o cc.so -buildmode=c-shared main.go

/usr/src/app # readelf -d cc.so

Dynamic section at offset 0x10cd10 contains 22 entries:

Tag Type Name/Value

0x0000000000000001 (NEEDED) Shared library: [libc.musl-x86_64.so.1]

0x0000000000000010 (SYMBOLIC) 0x0

0x000000000000000c (INIT) 0x42000

0x000000000000000d (FINI) 0x92ed9

0x0000000000000019 (INIT_ARRAY) 0xa2078

0x000000000000001b (INIT_ARRAYSZ) 8 (bytes)

0x000000006ffffef5 (GNU_HASH) 0x270

0x0000000000000005 (STRTAB) 0xa50

0x0000000000000006 (SYMTAB) 0x378

0x000000000000000a (STRSZ) 1026 (bytes)

0x000000000000000b (SYMENT) 24 (bytes)

0x0000000000000003 (PLTGOT) 0x10deb0

0x0000000000000002 (PLTRELSZ) 720 (bytes)

0x0000000000000014 (PLTREL) RELA

0x0000000000000017 (JMPREL) 0x41a00

0x0000000000000007 (RELA) 0xe58

0x0000000000000008 (RELASZ) 265128 (bytes)

0x0000000000000009 (RELAENT) 24 (bytes)

0x000000000000001e (FLAGS) SYMBOLIC BIND_NOW STATIC_TLS

0x000000006ffffffb (FLAGS_1) Flags: NOW NODELETE

0x000000006ffffff9 (RELACOUNT) 11040

0x0000000000000000 (NULL) 0x0

/usr/src/app # python test.py

Traceback (most recent call last):

File "test.py", line 2, in 

lib = ctypes.cdll.LoadLibrary('./cc.so')

File "/usr/lib/python2.7/ctypes/init.py", line 444, in LoadLibrary

return self._dlltype(name)

File "/usr/lib/python2.7/ctypes/init.py", line 366, in init

self._handle = _dlopen(self._name, mode)

OSError: Error relocating ./cc.so: : initial-exec TLS resolves to dynamic definition in ./cc.so

- show quoted text -

Code:

//main.go package main import "C" //export add func add(left, right int) int { return left + right } //export minus func minus(left, right int) int { return left - right } //export multiply func multiply(left, right int) int { return left * right } //export divide func divide(left, right int) int { return left / right } //export testPrint func testPrint(){ print("test") } func main() { } // test.py import ctypes lib = ctypes.cdll.LoadLibrary('./cc.so') if lib is not None: print ("can load so")

[ma...@eliasnaur.com]

Perhaps you're running into https://github.com/golang/go/issues/13492.

 - elias