Go shared libraries

[Elias Naur]

Hi,

During a review of my Android proposal, I was encouraged to separately describe the issues with support for shared libraries written in Go. The document is not exhaustive, so it is not a complete proposal. However, I hope it can serve to get the ball rolling. The document is a companion to my Android proposal, which if accepted as written, depends on some form of shared library support.

https://docs.google.com/document/d/1tU_yNmwu1raKZm9atHJ8THiQizkYTKykdWraIb15TSE

 - elias

[Michael Hudson-Doyle]

Hi,

Thanks for writing this.

Two motivations you don't mention are the ones that make distributions
favour shared libraries: saving disk space and ease of security updates
(e.g. being able to just update go.crypto rather than all applications
that are built with it).

I think it's also worth commenting that there is a way to get shared
libraries with Go currently: by using gccgo. So you can play there with
the performance impact of PIC code and so on. I've not tried it, but I
would expect that gccgo only supports loading go code into a go program
(or at least, one that has already loaded libgo.so -- maybe its .init
section sets everything up?).

Cheers,
mwh

[Elias Naur]

Hi Michael,

Thank you for your review.

On Tue, Feb 11, 2014 at 12:07 AM, Michael Hudson-Doyle <michael...@linaro.org> wrote:

Elias Naur <elias...@gmail.com> writes:

> Hi,
>
> During a review of my Android proposal, I was encouraged to separately
> describe the issues with support for shared libraries written in Go. The
> document is not exhaustive, so it is not a complete proposal. However, I
> hope it can serve to get the ball rolling. The document is a companion to
> my Android proposal, which if accepted as written, depends on some form of
> shared library support.
>
> https://docs.google.com/document/d/1tU_yNmwu1raKZm9atHJ8THiQizkYTKykdWraIb15TSE

Hi,

Thanks for writing this.

Two motivations you don't mention are the ones that make distributions
favour shared libraries: saving disk space and ease of security updates
(e.g. being able to just update go.crypto rather than all applications
that are built with it).

I think this point is indirectly covered by the observation that static linking is preferred in Go. For example, Rob specifically mentions the disk space argument in his critique of shared libraries here:

http://harmful.cat-v.org/software/dynamic-linking/

In other words, I don't think I'll get very far with motivations based on any inherent advantage of shared libraries.

I think it's also worth commenting that there is a way to get shared
libraries with Go currently: by using gccgo. So you can play there with

the performance impact of PIC code and so on.  I've not tried it, but I
would expect that gccgo only supports loading go code into a go program
(or at least, one that has already loaded libgo.so -- maybe its .init
section sets everything up?).

I know too little about gccgo to be able to write an accurate description of its shared library capabilities.

6l and 5l already implement PIC code - it was added as part of my prototype implementation for adding shared library support. So it would be relatively easy to measure the performance difference if necessary. But I doubt that a performance degradation and code size increase (however small) caused by a fringe feature like shared libraries would be acceptable.

 - elias

[Ian Lance Taylor]

Thanks. I don't have permission to comment on the doc, so I'll make
some notes here.

This doc goes into the issues before it clearly describes the goals.
What exactly do we want?

In the "Why?" section you suggest that shared libraries could be used
for plugins. Presumably if we support plugins to a Go program where
the plugin is written in Go we want to support passing channels to a
plugin. That implies certain choices for how to handle multiple
shared libraries.

When you talk about os.Args not being available to a library, that
suggests a model in which a Go shared library is being called from a
non-Go program. The Go shared library is going to be somehow loaded
into the process. In that mode, is it important to support a strict
dlopen/dlsym interface? For example, perhaps we can simply pass the
argument and environment to whatever function we use to load the
shared library. You mention the flags package explicitly, but does it
make sense to ever use the flags package in a mode where a Go shared
library is loaded into a non-Go program? Perhaps it's fine to crash
in that case.

It clearly doesn't make sense to call main.main when loading a Go
plugin into a Go program. And I don't think it makes sense to call
main.main when loading a Go shared library into a non-Go program. In
any case these choices have to be made based on how we want shared
libraries to behave.

In short this doc is taking an implementation-centric point of view,
which is understandable but insufficient. We need to develop a
goal-oriented point of view. We don't want to implement Go shared
libraries as whatever happens to work. We want to decide what will be
useful, and then adjust that based on what we can implement.

Ian

[Michael Jones]

Also good to discuss dynamic loading as purely accretive vs. load/unload.

--

---
You received this message because you are subscribed to the Google Groups "golang-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

Michael T. Jones | Chief Technology Advocate  | m...@google.com |  +1 650-335-5765

[Ian Lance Taylor]

On Mon, Feb 10, 2014 at 3:07 PM, Michael Hudson-Doyle
<michael...@linaro.org> wrote:
>
> I think it's also worth commenting that there is a way to get shared
> libraries with Go currently: by using gccgo. So you can play there with
> the performance impact of PIC code and so on. I've not tried it, but I
> would expect that gccgo only supports loading go code into a go program
> (or at least, one that has already loaded libgo.so -- maybe its .init
> section sets everything up?).

Gccgo supports splitting up a single Go program into multiple shared
libraries, but it doesn't support dynamically loading a shared
library. Only packages that are imported by main will be initialized.

For C/C++ code on x86_64 the PIC penalty is 2 to 3%. C/C++ permit
symbol interposition for any global symbol, a concept that doesn't
really make sense for Go. So in general I think it would be
reasonable to compile Go code to not use a PLT, and to use PC-relative
references to global variables in the same shared library rather than
using a GOT. Overall the PIC penalty ought to be less.

Ian

[Elias Naur]

On Tue, Feb 11, 2014 at 2:43 AM, Ian Lance Taylor <ia...@golang.org> wrote:

On Mon, Feb 10, 2014 at 5:20 AM, Elias Naur <elias...@gmail.com> wrote:
>
> During a review of my Android proposal, I was encouraged to separately
> describe the issues with support for shared libraries written in Go. The
> document is not exhaustive, so it is not a complete proposal. However, I
> hope it can serve to get the ball rolling. The document is a companion to my
> Android proposal, which if accepted as written, depends on some form of
> shared library support.
>
> https://docs.google.com/document/d/1tU_yNmwu1raKZm9atHJ8THiQizkYTKykdWraIb15TSE

Thanks.  I don't have permission to comment on the doc, so I'll make
some notes here.

Sorry, commenting should be enabled now.

 

This doc goes into the issues before it clearly describes the goals.
What exactly do we want?

I've changed the "Why" to a "Goals" section, describing the goal I find clear and easy to define: Loading Go libraries from non-Go programs allowing calls back and forth through the already existing cgo export facility. With that supported, it is a natural extension to allow Go programs to load Go libraries and communication through the very same Cgo interface. Cgo interfacing is limited, but well defined.

In the "Why?"  section you suggest that shared libraries could be used
for plugins.  Presumably if we support plugins to a Go program where
the plugin is written in Go we want to support passing channels to a
plugin.  That implies certain choices for how to handle multiple
shared libraries.

And this is where my understanding or imagination is perhaps too limited and the reason the Go-program/Go-library mode is so vaguely described in the document. 

The problem is that I don't see how to expand the interface between Go programs and Go libraries without imposing unbearable ABI requirements and version skew problems that reminds me of the notoriously unstable C++ ABI. You mention passing a channel, but what types are allowing over that channel? Are pointers allowed? It seems to me that allowing anything interesting being passed you might as well allow arbitrary Go function calls between program and library, leading to the ABI and versioning problems.

In short, I see the shared library facilities as too low level and generic to allow anything more powerful than the already existing Cgo interface.

Perhaps you (or someone else on this thread) could hint on how expanding the interface between two bodies of Go code might work?

 

When you talk about os.Args not being available to a library, that
suggests a model in which a Go shared library is being called from a
non-Go program.  The Go shared library is going to be somehow loaded
into the process.  In that mode, is it important to support a strict
dlopen/dlsym interface?  For example, perhaps we can simply pass the
argument and environment to whatever function we use to load the
shared library.

I'm not sure I understand you here, could you expand on this? In the general case, we don't have control over the non-Go program, in which case dlopen/dlsym is the way the programs loads and interface with our Go library.

 

You mention the flags package explicitly, but does it
make sense to ever use the flags package in a mode where a Go shared
library is loaded into a non-Go program?  Perhaps it's fine to crash
in that case.

It clearly doesn't make sense to call main.main when loading a Go
plugin into a Go program.  And I don't think it makes sense to call
main.main when loading a Go shared library into a non-Go program.  In
any case these choices have to be made based on how we want shared
libraries to behave.

Good points. I have different preferences, but it is pointless to argue over the details at this point.

 

In short this doc is taking an implementation-centric point of view,
which is understandable but insufficient.  We need to develop a
goal-oriented point of view.  We don't want to implement Go shared
libraries as whatever happens to work.  We want to decide what will be
useful, and then adjust that based on what we can implement.

I'm sorry the doc is unsatisfactory. I think the main obstacle for its usefulness is the Go-program/Go-library case, which I see as impractical to implement and inclined to leave as a non-goal. If you could allow me some pointers in a viable direction for that mode I'll gladly expand the document.

 - elias

[Elias Naur]

On Tue, Feb 11, 2014 at 2:50 AM, Michael Jones <m...@google.com> wrote:

Also good to discuss dynamic loading as purely accretive vs. load/unload.

I'm not sure I understand you. Could you expand on this? 

 - elias

[Ian Lance Taylor]

On Tue, Feb 11, 2014 at 2:23 AM, Elias Naur <elias...@gmail.com> wrote:
>
> The problem is that I don't see how to expand the interface between Go
> programs and Go libraries without imposing unbearable ABI requirements and
> version skew problems that reminds me of the notoriously unstable C++ ABI.
> You mention passing a channel, but what types are allowing over that
> channel? Are pointers allowed? It seems to me that allowing anything
> interesting being passed you might as well allow arbitrary Go function calls
> between program and library, leading to the ABI and versioning problems.

OK, I hadn't grasped this before. You are suggesting that we should
support shared libraries but only permit a cgo style interface when
calling them. That implies that each shared library will have its own
copy of the runtime. They can pass pointers back and forth, but each
call between the libraries (or the main program and a library) will be
through a defined and exported function. There will be no way to pass
a value to another shared library and have that library call a method
that calls back to the original library. It means that when passing a
pointer to another shared library, it will be essential to keep a
pointer live in the originating library, just as we must keep pointers
live when passing them to C code.

At a slightly higher level, for calling between Go shared libraries,
one can imagine building something like the net/rpc package.

> Perhaps you (or someone else on this thread) could hint on how expanding the
> interface between two bodies of Go code might work?

At the moment I'm much less interested in how we can do something than
in figuring out what we want to do. Once we know what we want to do,
we can talk about how to do it.

>> When you talk about os.Args not being available to a library, that
>> suggests a model in which a Go shared library is being called from a
>> non-Go program. The Go shared library is going to be somehow loaded
>> into the process. In that mode, is it important to support a strict
>> dlopen/dlsym interface? For example, perhaps we can simply pass the
>> argument and environment to whatever function we use to load the
>> shared library.
>
>
> I'm not sure I understand you here, could you expand on this? In the general
> case, we don't have control over the non-Go program, in which case
> dlopen/dlsym is the way the programs loads and interface with our Go
> library.

Any interface between a non-Go program and a Go shared library has to
be based on dlopen/dlsym but it doesn't have to be restricted to that.
For example, we could say that after calling dlopen the non-Go program
is required to use dlsym to call a specific function passing specific
arguments. Or we could say that the non-Go program is required to
provide certain functions that the Go shared library can call from its
initialization code. When you say that we don't have control over the
non-Go program, you are assuming a goal, and it is the goals that I
want to talk about: you are implying that we should make it possible
to write a Go shared library that can support an arbitrary plugin
interface of a non-Go program. That's a useful goal but let's make it
explicit.

>> In short this doc is taking an implementation-centric point of view,
>> which is understandable but insufficient. We need to develop a
>> goal-oriented point of view. We don't want to implement Go shared
>> libraries as whatever happens to work. We want to decide what will be
>> useful, and then adjust that based on what we can implement.
>>
>
> I'm sorry the doc is unsatisfactory. I think the main obstacle for its
> usefulness is the Go-program/Go-library case, which I see as impractical to
> implement and inclined to leave as a non-goal. If you could allow me some
> pointers in a viable direction for that mode I'll gladly expand the
> document.

I'm sorry to be so critical, and I really do appreciate you starting
the conversation. I'm just going to keep stressing that we need to
know what we want to do before we can discuss how we are going to do
it.

Ian

[Ian Lance Taylor]

The question is: do we support dlclose?

Ian

[Fredrik Ehnbom]

> I've changed the "Why" to a "Goals" section, describing the goal I find clear and easy to define: Loading Go libraries from non-Go programs allowing calls back and forth

IMO this is the key to what I'd personally would like to see. 

Here are a couple of concrete use cases besides the "Android apk" one that I'd like to have possible:

- Writing a Python module in Go rather than C/C++ (http://docs.python.org/2/extending/extending.html)

- Writing a Clang plugin in Go rather than C/C++ (http://clang.llvm.org/docs/ClangPlugins.html)

- Writing a Unity plugin in Go rather than C/C++ (http://docs.unity3d.com/Documentation/Manual/Plugins.html)

- Writing a VST plugin in Go rather than C/C++ (http://www.steinberg.net/en/company/developer.html)

From a use case perspective, the host application shouldn't need to know that the code I wrote was written in Go, some other language or a combination of multiple languages. Having both the host application and several plugins written in Go should work as a use case but I don't personally care about being able to use e.g. channels between the host and the plugin.

I think of the boundary between the host and the shared library as a "C boundary", where the exported api in the shared library looks just like C to anyone loading it and memory pointers passed between host and plugin would have to be handled as if either or both of the host and plugin was C.

/f

[Elias Naur]

On Tue, Feb 11, 2014 at 3:33 PM, Ian Lance Taylor <ia...@golang.org> wrote:

On Tue, Feb 11, 2014 at 2:23 AM, Elias Naur <elias...@gmail.com> wrote:
>
> The problem is that I don't see how to expand the interface between Go
> programs and Go libraries without imposing unbearable ABI requirements and
> version skew problems that reminds me of the notoriously unstable C++ ABI.
> You mention passing a channel, but what types are allowing over that
> channel? Are pointers allowed? It seems to me that allowing anything
> interesting being passed you might as well allow arbitrary Go function calls
> between program and library, leading to the ABI and versioning problems.

OK, I hadn't grasped this before.  You are suggesting that we should
support shared libraries but only permit a cgo style interface when
calling them.  That implies that each shared library will have its own
copy of the runtime.  They can pass pointers back and forth, but each
call between the libraries (or the main program and a library) will be
through a defined and exported function.  There will be no way to pass
a value to another shared library and have that library call a method
that calls back to the original library.  It means that when passing a
pointer to another shared library, it will be essential to keep a
pointer live in the originating library, just as we must keep pointers
live when passing them to C code.

Pretty much spot on. I'm sorry I didn't make that more clear.

 

At a slightly higher level, for calling between Go shared libraries,
one can imagine building something like the net/rpc package.

Sure, that's an option that avoids the ABI and version skew problems.

you are implying that we should make it possible
to write a Go shared library that can support an arbitrary plugin
interface of a non-Go program.  That's a useful goal but let's make it
explicit.

Great, that is exactly what I want. I've updated the goals section, using some of your description. How do we proceed from here?

The question is: do we support dlclose?

Thanks, I've added language for dlclose in the "Shutdown" section. I assume that we could start by not supported dlclose and then later, if needed, add support in a backwards compatible way.

 - elias

[Michael Jones]

On Tue, Feb 11, 2014 at 8:30 AM, Elias Naur <elias...@gmail.com> wrote:

The question is: do we support dlclose?

Thanks, I've added language for dlclose in the "Shutdown" section. I assume that we could start by not supported dlclose and then later, if needed, add support in a backwards compatible way.

Since the Go runtime owns the memory, channels, pending function calls (defers, closures), goroutines, and GC of any unloadable thing, there needs to be a way to handle all of this before, during, and after an unload. This is not a trivial request and may well involve some "before I call this function I need to check to see of it is still loaded" type of atomic double step before every function call. There are many other potentially significant ramifications as well. (Alternatively, clever memory mapping and trapping can be used to catch offenses of this kind but the problem remains of what to do when they happen.) It is not  simple matter. Yet, it does seem very, very useful to be able to unload modules if you're allowed to load them.

[Ian Lance Taylor]

On Tue, Feb 11, 2014 at 8:30 AM, Elias Naur <elias...@gmail.com> wrote:
>
> Great, that is exactly what I want. I've updated the goals section, using
> some of your description. How do we proceed from here?

OK, thanks. This set of goals makes sense to me.

I would like to hear whether other people have comments on this
approach.

Ian

[Gustavo Niemeyer]

The goal stated there definitely makes sense. One concern is that it
seems to omit the path towards having actual Go shared libraries. That
is, having a runtime that is able to load shared libraries for
packages or sets of packages. There are many well known use cases for
that; the one closest to my heart is enabling platforms (Linux
distributions, etc) to get core dependencies upgraded for fixing
security issues without having to rebuild everything.


gustavo @ http://niemeyer.net

[Elias Naur]

I specifically avoided general shared library support as a goal, in part because of the views of (some of) the core devs (and I tend to agree):

http://harmful.cat-v.org/software/dynamic-linking/

 - elias

[Gustavo Niemeyer]

I'm well aware of the rants, and I personally agree that the
importance of shared libraries is often overstated. With that said,
there are of course reasonable use cases for them, and as far as I
recall from previous conversations, the core team is actually
sympathetic to adding it in due time. So if we are going to implement
support for loading Go code as shared libraries, we might as well talk
about how we plan to implement the rest of it, even if we do these in
separate steps. One should lead into the other, rather than
conflicting with it.

As a side note, given the importance of being able to perform security
updates at a granularity that does not involve recompiling every
single binary in the system again, we can probably get either a
contract or a bounty via Canonical to someone that works with the core
team to get this sorted. If someone is interested, please get in
touch.

--

gustavo @ http://niemeyer.net

[gvdschoot]

I am probably going to make a fool out of myself with the following suggestion, but here it is anyway.

How about the Oberon approach?

That is each package compiled seperately and the compiled "module" placed in, let's say "/lib/go/package" such as "/lib/go/io.pkg", "/lib/go/runtime.pkg", "/lib/go/crypto/aes.pkg" and "/lib/go/3rd_party_package.pkg".

This is of course for Go programs only. It also probably only works on *nix, that is non-Windows.

The benefit is that *each* package is only there *once* so replacing it is done in *one* step with no interfering leftovers.

The security patch consists of a couple of lines of Go source code. Then the Go compiler recompiles and tests only that module.

Regarding the C shared libraries ".so"

What if there is a package "/lib/go/runtime_shared_C_lib.pkg"? (which should have a better name)

The .so shared library (written in Go) calls the C shared lib runtime and from there on everything is Go again.

This runtime does all the housekeeping and the .so file is "only" the interface to the C program that calls it and some logic.

The benefits are:

* Easy for anyone to understand
* Easy updating (source code only) and testing
* No code redundancy (no leftovers)
* Fast compilation (and specific)
* Small compiled modules (not sure that's a benefit)
* All the source code is available (let's say in "/lib/go/src/"
* Scales very well due namespaces
* Communicating with C in one middleware module only.
* No interference with statically linked Go programs.

Drawbacks:

* Yeah. Probably lots of them. But it is the idea that I wanted to vent because I didn't read it here.

[Geert-Johan Riemer]

On Tuesday, 11 February 2014 17:30:54 UTC+1, Elias Naur wrote:

The question is: do we support dlclose?

Thanks, I've added language for dlclose in the "Shutdown" section. I assume that we could start by not supported dlclose and then later, if needed, add support in a backwards compatible way.

 - elias

I think there are good use-cases for dlclose support. One of them being: plugins.

An example; I'm now working on adding Go support for UDF (User Defined Functions) in Aerospike DB.

I'm using go1.5beta2 to create the shared objects and let aerospike load them at runtime when a plugin is uploaded/registered to the aerospike database process.

It all works pretty well, except that I can't unload the plugins. So when a newer version of the plugin is registered to the database, I'll just have to overwrite references to the old plugin (leaving it unused, but still in memory).

This means that every update of a plugin costs resources and the only way to get rid of that is to completely restart the aerospike process, which causes downtime.

When I read go1.5 included .so build support, I assumed it would support dlclose. I know, assumptions assumptions...

Since dlclose probably won't land in go1.5, maybe the release notes should contain a warning about it. Or point to documentation about `-buildmode c-shared` with a warning in those.

I would very much like to see dlclose support in go1.5.1, but that's just my very opinionated wish ;)

[David Crawshaw]

I don't know if anyone will work on it, but I turned
golang.org/issue/11100 into a feature request for dlclose support. It
seems tricky to me: if multiple c-archives are opened, they share a
runtime. So there will have to be some kind of accounting in the
runtime to make sure it doesn't shut down until all c-archives are
closed.

For 1.5, you could consider using separate processes for your UDFs. I
have done something similar before. The downside is you have to copy
data to the other process, the upside is you can restrict the CPU and
memory use of the subprocess.

> --

> You received this message because you are subscribed to the Google Groups
> "golang-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-dev+...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

[demetri...@gmail.com]

Note that you can use shared memory for the IPC.