Re: [PATCH] RFC: go shared libraries support (Linux/x64)

[bryanturley]

What happens when you run two go based .so files?
If they are both making threads thinking they are the only go runtime it could lead to some interesting chaos.

Also this probably should have been posted on golang-dev I think.

[Elias Naur]

I've deleted the original post)I've and replied to this in the cross-post in golang-dev. Sorry for the inconvenience.

 - elias

[Rémy Oudompheng]

On 2012/11/3 Elias Naur <elias...@gmail.com> wrote:
> Hi,
>
> Eventually, I’d like to write shared libraries in go, in particular I want
> my go code callable from the android dalvik virtual machine. So, after a
> long and intense staring contest with the tools and the go runtime, I have a
> proof of concept running under linux/x64. I know that your contribution
> guide mentions that I should write my design before coding, but in this case
> I chose to do the experiments myself first to get a feel of the go code and
> to figure out if it was even possible for me to implement.

At first I thought the change was about turning Go packages into
shared libraries but that's not what it is about, it's about embedding
a Go program/library into an external program.

Rémy.

[Elias Naur]

That's what I thought, at least. Apparently I need to go through moderation before my replies appear. 

 - elias

[bryanturley]

Also this probably should have been posted on golang-dev I think.

I've deleted the original post)I've and replied to this in the cross-post in golang-dev. Sorry for the inconvenience.

 - elias

Meh, don't think it was that big of a deal, or even a deal at all really.

[Elias Naur]

Exactly. 

 - elias

[minux]

On Sun, Nov 4, 2012 at 12:27 AM, Elias Naur <elias...@gmail.com> wrote:

Eventually, I’d like to write shared libraries in go, in particular I want my go code callable from the android dalvik virtual machine. So, after a long and intense staring contest with the tools and the go runtime, I have a proof of concept running under linux/x64. I know that your contribution guide mentions that I should write my design before coding, but in this case I chose to do the experiments myself first to get a feel of the go code and to figure out if it was even possible for me to implement.

Please take a look at http://code.google.com/p/go/issues/detail?id=4069. 

I think that's the route to go for producing dynamic libraries.

btw, according to contribution guide, this thread does belong to golang-nuts.

[Elias Naur]

I've seen that, but I believe that my work is a subset of that solution, since you still need PIC and runtime support to make .o files useful for gcc linking into shared libraries (.so).

 

btw, according to contribution guide, this thread does belong to golang-nuts.

Oh my, what a mess :) golang-nuts is best for me anyway, because I currently don't have posting ability in golang-devl. For reference, the golang-dev thread is at:

https://groups.google.com/d/topic/golang-dev/M8w4TolblDg/discussion

 - elias 

[Patrick Mylund Nielsen]

Now everybody knows, at least :)

 - elias 

--
 
 

[Elias Naur]

On Saturday, November 3, 2012 5:36:54 PM UTC+1, bryanturley wrote:

What happens when you run two go based .so files?
If they are both making threads thinking they are the only go runtime it could lead to some interesting chaos.

I haven't tried to address that issue, but I suspect that with proper hiding of the shared library symbols, two go shared libraries could happily coexist, with two seperate runtimes, blissfully unaware of each other. Not the most efficient solution, but certainly simpler that trying to let two go libs share a single, global go runtime.

 - elias

[minux]

On Sun, Nov 4, 2012 at 1:01 AM, Elias Naur <elias...@gmail.com> wrote:

I haven't tried to address that issue, but I suspect that with proper hiding of the shared library symbols, two go shared libraries could happily coexist, with two seperate runtimes, blissfully unaware of each other. Not the most efficient solution, but certainly simpler that trying to let two go libs share a single, global go runtime.

consider sole reference to memory allocated from one runtime passed to

Go code running under the other runtime, the first runtime will consider

the memory garbage, but it's used in fact.

This is part of the problem why Go doesn't produce dynamic library

now.

Another complexity is, Go never guarantee stable ABI, with static linking,

this doesn't matter; but with dynamic linking, we will need some kind of

versioning system to prevent from loading ABI-incompatible libraries.

(yes, i know ELF has symbol versioning and soname mechanism, but

what about Mach-O and PE/COFF?)

[Elias Naur]

On Sat, Nov 3, 2012 at 6:49 PM, minux <minu...@gmail.com> wrote:

On Sun, Nov 4, 2012 at 1:01 AM, Elias Naur <elias...@gmail.com> wrote:

I haven't tried to address that issue, but I suspect that with proper hiding of the shared library symbols, two go shared libraries could happily coexist, with two seperate runtimes, blissfully unaware of each other. Not the most efficient solution, but certainly simpler that trying to let two go libs share a single, global go runtime.

consider sole reference to memory allocated from one runtime passed to

Go code running under the other runtime, the first runtime will consider

the memory garbage, but it's used in fact.

How is that different from the problem of passing references to C code through cgo? AFAIK, the proper way to do that is to hold a reference to the data on the go side while it is accessed from C.

 

This is part of the problem why Go doesn't produce dynamic library

now.

Another complexity is, Go never guarantee stable ABI, with static linking,

this doesn't matter; but with dynamic linking, we will need some kind of

versioning system to prevent from loading ABI-incompatible libraries.

(yes, i know ELF has symbol versioning and soname mechanism, but

what about Mach-O and PE/COFF?)

Do you mean in the one-global-runtime case? If two go libraries are completely seperate with seperate runtimes, schedulers etv. you shouldn't need ABI stability other than what the C interface gives you.

 - elias

[minux]

On Sun, Nov 4, 2012 at 1:54 AM, Elias Naur <elias...@gmail.com> wrote:

On Sat, Nov 3, 2012 at 6:49 PM, minux <minu...@gmail.com> wrote:

On Sun, Nov 4, 2012 at 1:01 AM, Elias Naur <elias...@gmail.com> wrote:

I haven't tried to address that issue, but I suspect that with proper hiding of the shared library symbols, two go shared libraries could happily coexist, with two seperate runtimes, blissfully unaware of each other. Not the most efficient solution, but certainly simpler that trying to let two go libs share a single, global go runtime.

consider sole reference to memory allocated from one runtime passed to

Go code running under the other runtime, the first runtime will consider

the memory garbage, but it's used in fact.

How is that different from the problem of passing references to C code through cgo? AFAIK, the proper way to do that is to hold a reference to the data on the go side while it is accessed from C.

This problem is vastly different. Passing pointer to cgo is crossing language borders, but

passing Go memory to Go (under another runtime) shouldn't be regarded as crossing

language borders, and should be fully transparent to the user.

Consider the the most common use case of dynamic loading Go code: dynamic plugin.

do you really think that the user must make sure the main application retain every

pointer passed to the plugin? Then we've returned to the old way of manual memory

management, and in sometimes, this won't even be possible.

 

This is part of the problem why Go doesn't produce dynamic library

now.

Another complexity is, Go never guarantee stable ABI, with static linking,

this doesn't matter; but with dynamic linking, we will need some kind of

versioning system to prevent from loading ABI-incompatible libraries.

(yes, i know ELF has symbol versioning and soname mechanism, but

what about Mach-O and PE/COFF?)

Do you mean in the one-global-runtime case? If two go libraries are completely seperate with seperate runtimes, schedulers etv. you shouldn't need ABI stability other than what the C interface gives you.

Yes. IMHO, one runtime is a must (of course, you can embed one runtime into each

dynamic library, but only one of them can be in charge at any time).

however, embedding one runtime into each dynamic Go library defeated the space

saving for dynamic loading.

[Elias Naur]

This is the reason we're discussing this at all :) I'm perfectly happy with statically linked stand-alone executables for my go programs, and not particularly interested in the space savings in disk space or runtime memory. My focus is the cases where a shared library is an absolute requirement. A great example is the android dalvik java runtime which _requires_ external native code to be in the form of a shared library. But anytime you'd like to write go plugins of any sort you'll very likely need dynamic library support in some form.

How multiple plugins written in go coexist is definitely an issue, but I'm willing to go with "unsupported" in the short run, "works, but inefficient in disk and memory space" in the longer run and some kind of "runtime sharing of code/runtime/memory" for the longest run.

But for now, I'd just like to get the basics working - position independent code, runtime changes, compiler flags etc which is exactly what my CLs address. As far as I see, you can't avoid that work, however you want to support go shared libraries in the future.

 - elias

[bryanturley]

Ooops.   My bad ;)

[bryanturley]

This is the reason we're discussing this at all :) I'm perfectly happy with statically linked stand-alone executables for my go programs, and not particularly interested in the space savings in disk space or runtime memory. My focus is the cases where a shared library is an absolute requirement. A great example is the android dalvik java runtime which _requires_ external native code to be in the form of a shared library. But anytime you'd like to write go plugins of any sort you'll very likely need dynamic library support in some form.

It is incredibly annoying that the android team wants everyone to use java only, then lets you load just about any random .so inside your java program.
I want to use go on android as well, and I know I have seen other threads to that end.
I wonder how many other languages aren't represented on android because of this decision.
The answer seems simple, FIX ANDROID, but that doesn't seem likely.

Sorry, I may have some pent up anger at the android team.... ;)

How multiple plugins written in go coexist is definitely an issue, but I'm willing to go with "unsupported" in the short run, "works, but inefficient in disk and memory space" in the longer run and some kind of "runtime sharing of code/runtime/memory" for the longest run.

But for now, I'd just like to get the basics working - position independent code, runtime changes, compiler flags etc which is exactly what my CLs address. As far as I see, you can't avoid that work, however you want to support go shared libraries in the future.

 - elias

 go build -dirty-android-hack
 I would use it.
 I should prob dig into the go code deeper myself see if I can help.

[Elias Naur]

--
 

Android is an especially annoying use case, since go already runs on android as stand alone executables, they're just no distributable as standard .apk files through Play and the like. Only way to do that is to run an embedded go program from java and communicate with it with some kind of IPC.

However, there are still many other cases where you have a main program extensible with plugins, and that is where shared libraries is the most common way to do that. Another useful case is where you have a "friendly" program that is either written entirely in Go or simply wants Go as a "scripting language" not unlike what Python is today. In that case, I could imagine shared library support would be be useful.

 - elias

[bryanturley]

Android is an especially annoying use case, since go already runs on android as stand alone executables, they're just no distributable as standard .apk files through Play and the like. Only way to do that is to run an embedded go program from java and communicate with it with some kind of IPC.

However, there are still many other cases where you have a main program extensible with plugins, and that is where shared libraries is the most common way to do that. Another useful case is where you have a "friendly" program that is either written entirely in Go or simply wants Go as a "scripting language" not unlike what Python is today. In that case, I could imagine shared library support would be be useful.

 - elias

Yes I am aware, I have written pure c programs for android just not inside of a .apk.  I have written some small toys with the NativeActivity wrapper garbage as well.
What I can't understand is why won't they just allow fully native code from a .apk, they could keep their all programs have a different uid scheme and permissions xml data and what not and just run a binary instead of a some java that loads a native .so.
It is a baffling.

[Greg Ward]

On 03 November 2012, Elias Naur said:
> Eventually, I’d like to write shared libraries in go, in particular I want
> my go code callable from the android dalvik virtual machine. So, after a
> long and intense staring contest with the tools and the go runtime, I have
> a proof of concept running under linux/x64.

Nifty!

> With the two CLs applied, building the runtime with PIC support is done with
>
> GO_FLAGS=-lib ./make.bash

Bikeshedding, since I'm completely ignorant of all the low-level
nitty-gritty details required to make this work ... but I do know how
to use GCC to build shared libraries from C, and the relevant option
there is -shared. So why not -shared instead of -lib?

Greg

[Elias Naur]

On Wed, Nov 7, 2012 at 4:12 AM, Greg Ward <gr...@gerg.ca> wrote:

> With the two CLs applied, building the runtime with PIC support is done with

>
> GO_FLAGS=-lib ./make.bash

Bikeshedding, since I'm completely ignorant of all the low-level
nitty-gritty details required to make this work ... but I do know how
to use GCC to build shared libraries from C, and the relevant option
there is -shared. So why not -shared instead of -lib?

       Greg

Sensible bikeshedding - I've updated the CLs to use -shared instead of -lib. I've also sync'ed to tip and verified all.bash still passes.

 - elias

[bryanturley]

Have you tried it on arm yet?

[Elias Naur]

On Wed, Nov 7, 2012 at 6:00 PM, bryanturley <bryan...@gmail.com> wrote:

Have you tried it on arm yet?

Not yet. Obviously ARM support is required for go Android libraries to work, but I started with x64 to get the generic bits done and because of familarity of the architecture and for convenience (I don't have access to an ARM based machine other than my phone).

At the moment, I'm holding my breath for some indication on whether my CLs are the right direction to go before I spend more time on the 5* arch specific parts.

 - elias

[Elias Naur]

Hi,

It's been a while, so I hg sync'ed and made a few bug fixes to my changes to it work with the current tip again. hg change <CL Id> didn't update the files for me, so I created a new CL at 6926049. Any pointers as to how to update the CL patch set would be much appreciated. Was it because I didn't change the description? Changes:

- a missing -lib to -shared conversion in cmd/go/testflags.go

- Removed unused typedef from cmd/6l/l.h

- moved rel_ro assignment into -U flag conditional in cmd/ld/data.c

- synced to tip and fixed, all tests again pass with or without GO_FLAGS=-shared

Additionally, I updated the test case to not hardcode paths. Download it, unzip it and run it with:

GOPATH=<absolute path to test dir> GOROOT=<path to go root built with GO_FLAGS=-shared> ./build.sh && LD_LIBRARY_PATH=. ./main

And lastly, I copied the original post below, since it was deleted by me because of confusion as to whether the post belonged in golang-dev.

- elias

Original post:

Eventually, I’d like to write shared libraries in go, in particular I want my go code callable from the android dalvik virtual machine. So, after a long and intense staring contest with the tools and the go runtime, I have a proof of concept running under linux/x64. I know that your contribution guide mentions that I should write my design before coding, but in this case I chose to do the experiments myself first to get a feel of the go code and to figure out if it was even possible for me to implement.

With the two CLs applied, building the runtime with PIC support is done with

GO_FLAGS=-shared ./all.bash

after which you can run the attached test case like this:

GOPATH=<absolute path to test dir> GOROOT=<path to go root built with GO_FLAGS=-shared> ./build.sh && LD_LIBRARY_PATH=. ./main

which should give the following output:

Main, calling cgo From test, calling go
Go callback!
C main, calling Go_callback
Go callback!
C main exiting

The test case demonstrates a C main program loading a go shared library (libgo.so), which in turn loads a C shared library (libtest.so). The test case calling sequence is as follows:

1. libgo.so DT_INIT is run by the dynamic linker. Go is initialized and main.main() is run. In this case, libtest.so is called from main.main through cgo.
2. main.main() returns and instead of exiting, control is relinquished back to the dynamic linker which in turn calls the C main().
3. C main() calls back into libgo.so through cgo and exits.

The tool changes (http://codereview.appspot.com/6775101) adds the new “-lib” flag to the go tool and the -U flags to 6l, 6g and 6c. With -U the 6g and 6c tools outputs code compatible with position independent code. 6l in turn outputs the necessary relocations and uses RIP-relative addressing in the resulting machine code. It also sets up DT_INIT to initialize the library and it marks the resulting elf file as a shared library instead of a regular executable. The resulting file is an elf shared library (ET_DYN) and can be used both as a executable and as a shared library. When run as an executable, there is the added benefit of activating ASLR since the file is recognized as position independent.

The runtime changes (http://codereview.appspot.com/6822078) adds runtime support for go shared libraries. In particular:

1. Assembler changes to allow 6l to output RIP-relative addressing on x64.
2. Scheduler changes to allow the main go routine to exit cleanly and relinquish control back to the main program (proc.c).
3. Changes to the stack unwinding code to compensate for the runtime relocations (symtab.c).
4. Changes the VDSO loader to use /proc/maps instead of the ancillary vector which is not available from the library init (vdso_linux_amd64.c and runtime.c)

I don’t expect the changes to go in as is, but I would very much welcome reviews, comments etc., to allow for go shared library support, eventually on the other platforms and archs too.

 - elias

[Ian Lance Taylor]

On Tue, Dec 11, 2012 at 7:27 AM, Elias Naur <elias...@gmail.com> wrote:
>
> It's been a while, so I hg sync'ed and made a few bug fixes to my changes to
> it work with the current tip again. hg change <CL Id> didn't update the
> files for me, so I created a new CL at 6926049. Any pointers as to how to
> update the CL patch set would be much appreciated. Was it because I didn't
> change the description?

You shouldn't need to update the description. You should be able to
just change the list of files. "hg pending" should show the CLs in
your working directory, and also list the files that are not in any
CL. Once you update the CL in your working directory, "hg upload
NNNNNN" will upload it to the codereview site.

Ian

[Elias Naur]

Hi Ian,

Thank you. Yes, Russ also pointed me to "hg upload" which was the command I needed. I can see that the contribution guide lists "hg mail" which apparently does the same plus mails the CL to reviewers, but I didn't want to bother reviewers. If there's not a list of commands in the mercurial extension elsewhere, maybe hg upload and the rest of the mercurial extension commands should also be listed in the guide?

 - elias

[Russ Cox]

'hg help codereview' will show the codereview commands. I will add a
note about that to the web page.

[Elias Naur]

I've updated the changesets to the latest revision and verifed everything works. Minor change:

 

 - The tools CL (6926049) fixed a small symbol generation bug, which was needed for shared libraries to work. A slightly different fix for the same bug was recently fixed by minux, so I've integrated that in 6926049.

Russ and everyone else: Will it make any difference to "hg mail" the changesets or will it simply annoy?

 - elias

[Mike Rosset]

you should hg mail. since that will mail to golang-dev.

> --
>
>

[Mike Rosset]

Elias did you mail this patch off? I cant seem to add comments to it.
I was wondering what the status was.

[minux]

On Monday, December 24, 2012, Mike Rosset wrote:

Elias did you mail this patch off? I cant seem to add comments to it.

you can add and mail comments even the CL is not mailed out.

note to the cl author: if you don't want to automatically mail out

review request for the CL

as a side effect of replying to comments, please reply from your

email client, not from reitveld.

[Elias Naur]

Hi Mike,

Yes, I've now hg mail'ed the CLs

 - elias

[Kevin Malachowski]

Is there any recent progress on this? I can't seem to figure out how to get the -shared flag to work for 6l. It complains with the following:

"entry not text: _rt0_amd64_linux_lib
main.main: doasm: notfound from=78 to=8f (7)    MOVQ    $type.string+0(SB),(SP)
(A whole bunch of other errors)
"

And all the program does is import "fmt" and tries to Println a simple string. go version reports

"go version devel +08d20469cc20 Tue Mar 26 08:27:18 2013 +0100 linux/amd64"

And I'm on an Ubuntu 12.04 virtual machine.

In addition, when I try to use the -shared flag with `go build` it fails, complaining that the -shared flag was provided but not defined.

[Elias Naur]

You'll need to pass the -largemodel flag to the Go and C compiler too, to avoid assembler instructions that can't use position independent addressing. However, the shared library work still need the runtime changes that were never merged. If nobody else beats me to it, I plan to update the work as soon as the current linker and cgo changes have settled down, some time after 1.1.

The planning work for the semantic changes for the runtime is tracked here:

https://code.google.com/p/go/issues/detail?id=4848

Host linker support (issue 4069, complete for linux/darwin and i386/x64) and support for cgo callbacks from foreign threads (issue 4435, fixed) were recently added. With that, my best bet is splitting Go initialization to run separately, and only if Go is the main program, run main.main(). Other than that, there are various nitty gritty details to handle, as hinted in 4848 (missing os.Args being one)

 - elias

[小菜]

Don't understand！

在 2012年11月4日星期日 UTC+8上午12:36:54，Bryan Turley写道：

What happens when you run two go based .so files?
If they are both making threads thinking they are the only go runtime it could lead to some interesting chaos.

Also this probably should have been posted on golang-dev I think.

[Bryan Turley]

There was a message before that one. Elias Naur was showing his go on android work.
This later evolved into the current go on android stuff.