about -buildmode c-shared

[Frédéric De Jaeger]

Hi, this is a followup of https://groups.google.com/g/golang-dev/c/J7W3sDexK18

that I've mistakenly posted in the wrong channel.  I feel ashamed about it.

Ian Lance Taylor wrotes:

> This is a consequence of the fact that (on most systems) shared
libraries share a symbol namespace, so that all references to, say,
runtime.chansend are resolved to the same function. If different Go
shared libraries built using -buildmode=c-shared are built with
different versions of Go, then the fact that they call the same
runtime.chansend function can mean that one of the shared libraries
will not work correctly.

Thanks for your quick response.

When I build a package like this

go build -work -buildmode c-shared -o pouet.so .

I get a standalone .so that contains anything needed (business code + go runtime)

The contract is that symbols like this:

//export XPluginStart
func XPluginStart(outName, outSig, outDesc *C.char) C.int {

are exported and made visible to the outside world (the C based host programm that loads the .so)

When I do:

> nm pouet.so | grep " T "

It gives many  publicly visible go symbols

00000000000d75a0 T x_cgo_callers
00000000000d71d0 T x_cgo_init
00000000000d7380 T x_cgo_mmap
00000000000d73b0 T x_cgo_munmap

and also mine:

00000000000d6be0 T XPluginStart

My understanding of that is that you can control the visibility of symbols, can't you ?

IMHO, all the other symbols (including everything from the runtime) could be hidden.

> That's not how shared libraries work on many systems.

I suppose I might be missing something.  I'm not a total expert on that matter. Suppose we tune the visibility of symbols such that the previous command would only shows the ones I specifically want to be accessible to the C host.  And everything else (runtime global variable, functions, ...) is made private (it appears with the symbol `t` instead of `T`).  Wouldn't that guarantee that successive loading of plugin like these would all have their own copy of the runtime ?

MacOS, linux, and windows all provide a way to control the symbols visibility.  Were you talking about other systems where you can't control the symbol visibility so easily ?

MacOS has his _two level namespace_ thingy.  It is supposed to guarantee (if I understand that stuff correctly) that symbol resolution is done right, (if `a.dylib` calls `foo` that exists in `a.dylib` and also in another `b.dylib`, the one in `a.dylib` gets called)  So what currently happens on MacOS ?  Don't we have several runtime instantiated in this scenario ?

I can imagine other traps on that road.  Other form of global per process resources could cause issues.  Signal, maybe ?

Fred

[Ian Lance Taylor]

On Sun, Feb 7, 2021 at 7:02 PM Frédéric De Jaeger
<fdej...@novaquark.com> wrote:
>
> Hi, this is a followup of https://groups.google.com/g/golang-dev/c/J7W3sDexK18
> that I've mistakenly posted in the wrong channel. I feel ashamed about it.

No need to feel ashamed, it's an easy mistake to make, and a harmless one.

Thanks, you may be correct: it may be possible to hide all symbols
other than the ones that are intended to be exported to the C program.
If it is possible to do that reliably in all cases, then the effect
should be that if you link against multiple shared libraries, each
will have an independent Go runtime. The program will wind up with
different Go heaps and different garbage collectors competing with
each other. The result will be inefficient, sometimes alarmingly so,
but I don't immediately see any reason why it wouldn't work.

Ian

[Frédéric De Jaeger]

On Tuesday, February 9, 2021 at 2:46:33 AM UTC+1 Ian Lance Taylor wrote:

Thanks, you may be correct: it may be possible to hide all symbols
other than the ones that are intended to be exported to the C program.
If it is possible to do that reliably in all cases, then the effect
should be that if you link against multiple shared libraries, each
will have an independent Go runtime. The program will wind up with
different Go heaps and different garbage collectors competing with
each other. The result will be inefficient, sometimes alarmingly so,
but I don't immediately see any reason why it wouldn't work.

No bad surprise expected from other kind of uniqu per process resources ?  (signal? ...)

I suppose you mean other kind of inefficiency than the basic ram usage ?  I've observed that when I run several CPU intensive go apps on the same host, the cumulative  GC pause (Memstats.PauseTotalNs) raises dramatically (not linear in the number of go process running).  But in more gentle scenario, where the go plugin rarely runs, do we hit those inefficiencies ?

 

At the moment, go can't be used reliably as a tool to write C plugin for a generic C host.  Because of the runtime constraints, and the unloading issue.

I was naively assuming that the unloading issue is easy to tackle when the runtime is not shared between .so.  Is it true ? 

I can see a difficulty when there are running go routines doing cgo invocations (and we can probably fail/abort/crash in this case) .  Apart from that, I have the feeling this should be easy to cleanup all the runtime ressources (memory, threads).  I suppose this kind of cleanup logic was never written because exit(0) does the same job.

IMHO, being able to write generic C plugin in go, would be a good selling feature (I would appreciate it as much as all the people  here https://github.com/golang/go/issues/11100). 

So, suppose we implement that feature by hiding/unsharing the runtime (assuming it works and we fill in the missing pieces)

The question is how bad would be that solution compared to  the absence  of that feature ?

Are there really some business usage of the current existing feature ?  (people really loading several `c-shared` go .so and expecting to share the runtime)

What about yet another build mode like `c-shared-private-runtime` ?  unloading would be only implemented with this build mode.

A last technical question, do you link the shared object with the ld flag `- z nodelete`  (which turn dlclose into no op). 

[Ian Lance Taylor]

On Sat, Feb 13, 2021 at 1:53 AM Frédéric De Jaeger
<fdej...@novaquark.com> wrote:
>
> On Tuesday, February 9, 2021 at 2:46:33 AM UTC+1 Ian Lance Taylor wrote:
>>
>>
>>
>> Thanks, you may be correct: it may be possible to hide all symbols
>> other than the ones that are intended to be exported to the C program.
>> If it is possible to do that reliably in all cases, then the effect
>> should be that if you link against multiple shared libraries, each
>> will have an independent Go runtime. The program will wind up with
>> different Go heaps and different garbage collectors competing with
>> each other. The result will be inefficient, sometimes alarmingly so,
>> but I don't immediately see any reason why it wouldn't work.
>>
> No bad surprise expected from other kind of uniqu per process resources ? (signal? ...)
>
> I suppose you mean other kind of inefficiency than the basic ram usage ? I've observed that when I run several CPU intensive go apps on the same host, the cumulative GC pause (Memstats.PauseTotalNs) raises dramatically (not linear in the number of go process running). But in more gentle scenario, where the go plugin rarely runs, do we hit those inefficiencies ?

Probably not. I would still be concerned about surprising behavior.

> At the moment, go can't be used reliably as a tool to write C plugin for a generic C host. Because of the runtime constraints, and the unloading issue.

Yes.

> I was naively assuming that the unloading issue is easy to tackle when the runtime is not shared between .so. Is it true ?

No. The current Go runtime has no ability to shut down all
goroutines. If any goroutines are left running, and the plugin is
unloaded, the program will crash.

> I can see a difficulty when there are running go routines doing cgo invocations (and we can probably fail/abort/crash in this case) . Apart from that, I have the feeling this should be easy to cleanup all the runtime ressources (memory, threads). I suppose this kind of cleanup logic was never written because exit(0) does the same job.

Yes, goroutines currently calling into C code are a particular
problem, but they are not the only problem.

> IMHO, being able to write generic C plugin in go, would be a good selling feature (I would appreciate it as much as all the people here https://github.com/golang/go/issues/11100).
> So, suppose we implement that feature by hiding/unsharing the runtime (assuming it works and we fill in the missing pieces)
> The question is how bad would be that solution compared to the absence of that feature ?
>
> Are there really some business usage of the current existing feature ? (people really loading several `c-shared` go .so and expecting to share the runtime)
> What about yet another build mode like `c-shared-private-runtime` ? unloading would be only implemented with this build mode.

I tend to agree that if people want to use Go code as plugins to a C
program, they would expect those Go plugins to operate independently.
But I think that many people would be disappointed by the effects on
resource usage, which they would have no way to mitigate. So we would
have a feature that would make people unhappy in practice.

> A last technical question, do you link the shared object with the ld flag `- z nodelete` (which turn dlclose into no op).

As far as I know we do not.

Ian

[Frédéric De Jaeger]

On Saturday, February 13, 2021 at 4:21:34 PM UTC+1 Ian Lance Taylor wrote:

> I was naively assuming that the unloading issue is easy to tackle when the runtime is not shared between .so. Is it true ?

No. The current Go runtime has no ability to shut down all
goroutines. If any goroutines are left running, and the plugin is
unloaded, the program will crash.

I was not assuming a synchronous uncooperative way for the runtime to kill a running goroutine.  I was more thinking about some cooperative strategy. Like a running go routine could check sometimes (maybe in the function prologue, or when doing an allocation) that it is time to die. 

Anycase, that could be also a constraint of the business code in the plugin.  Like in C, they are not supposed to be actively running when the .so is dlclosed.

In fact, what happens in practice today if we dlclose a plugin that has no running goroutine (and also no cgo invocations).  Does it "work" somehow ?  Are we leaking threads, waiting on some dead semaphore or something ?

> A last technical question, do you link the shared object with the ld flag `- z nodelete` (which turn dlclose into no op).

As far as I know we do not.

This `ld` flag makes me think of another approach.  Maybe it is  ok to just turn off the effective unloading of the .so (with that ld flag).  So dlclose is no-op.  But if the business code takes care of cleaning most of its global state, maybe the gc has the opportunity to return some memory back to the OS (well, since the gc is not moving stuff, maybe the hope is slim, I don't know).

That approach does not solve the problem of the developer iteratively working on a plugin.  Apart from that, this looks like an acceptable compromise for someone who wants to ship an loadable plugin that can be "unloaded".  At least it works somehow, leaking a bit of memory.  But it does not crash.

Of course, it is expected that if I load libfoo.so unload it, and load it again, everything works.  I don't see why it would not.  The second dlopen is presumably no-op.

Now about the unique  runtime constraint.  In the previous message, you said that we might be able to run several runtimes in the same process (private runtime symbols, ...)

If that works, it would mean that if the runtime symbols are public, but versioned, then maybe the runtime constraint is gone, isn't it? 

If two lib libfoo.so and libbar.so are built with different runtimes, we get two instantiated runtime in the binary that loads them simultaneously.  Well, too bad, we lose a bit of memory, but at least it works.  It looks acceptable to me.

In the favorable case, they share the same runtime and every one is happy.

I am curious about what happens on MacOS today.  Because my understanding of their two level namespace  (https://developer.apple.com/library/archive/documentation/Porting/Conceptual/PortingUnix/compiling/compiling.html#//apple_ref/doc/uid/TP40002850-BCIHJBBF) implies that you instantiate a new runtime whenever you load a c-shared .dylib.  I'm not sure at all.  It seems there is a way to force a flat namespace (it does not seem to be the default, thought).  I haven't verified that yet.  By the way, what the most efficient way to detect if two runtime are being instantiated ?  Something like a debug log that is supposed to happen once, or a similar trick.

Thanks

Fred

[Ian Lance Taylor]

On Mon, Feb 15, 2021 at 6:51 AM Frédéric De Jaeger
<fdej...@novaquark.com> wrote:
>
> On Saturday, February 13, 2021 at 4:21:34 PM UTC+1 Ian Lance Taylor wrote:
>>
>>
>>
>> > I was naively assuming that the unloading issue is easy to tackle when the runtime is not shared between .so. Is it true ?
>>
>> No. The current Go runtime has no ability to shut down all
>> goroutines. If any goroutines are left running, and the plugin is
>> unloaded, the program will crash.
>>
>
> I was not assuming a synchronous uncooperative way for the runtime to kill a running goroutine. I was more thinking about some cooperative strategy. Like a running go routine could check sometimes (maybe in the function prologue, or when doing an allocation) that it is time to die.

I don't see how that could be completely reliable, especially when Go
code calls into C code. For something like this Go aims for solutions
that are 100% reliable.

> Anycase, that could be also a constraint of the business code in the plugin. Like in C, they are not supposed to be actively running when the .so is dlclosed.
> In fact, what happens in practice today if we dlclose a plugin that has no running goroutine (and also no cgo invocations). Does it "work" somehow ? Are we leaking threads, waiting on some dead semaphore or something ?

The current Go plugin support does not support calling dlclose on a plugin.

> This `ld` flag makes me think of another approach. Maybe it is ok to just turn off the effective unloading of the .so (with that ld flag). So dlclose is no-op. But if the business code takes care of cleaning most of its global state, maybe the gc has the opportunity to return some memory back to the OS (well, since the gc is not moving stuff, maybe the hope is slim, I don't know).

The Go runtime does already return some memory to the operating system
over time. It won't return all of it, though.

> By the way, what the most efficient way to detect if two runtime are being instantiated ? Something like a debug log that is supposed to happen once, or a similar trick.

I don't know of a way to do this.

Ian

[Peter W]

Hi guys,

I am trying to load, call and unload multiple go libs which are built with -buildmode c-shared in a c problem, it’s doable on windows, Linux and macOS ? Also I see -buildmode shared removal was rejected, what about -buildmode archive?

Happy holidays!

[Ian Lance Taylor]

On Sun, Dec 26, 2021 at 3:45 PM Peter W <peter....@gmail.com> wrote:
>
> I am trying to load, call and unload multiple go libs which are built with -buildmode c-shared in a c problem, it’s doable on windows, Linux and macOS ? Also I see -buildmode shared removal was rejected, what about -buildmode archive?

You can't safely unload a Go DLL built with -buildmode=c-shared. That
is not supported at all. Sorry.

Loading multiple Go DLLs built with -buildmode=c-shared ought to work
on ELF based systems like Linux. I don't know whether it will work on
Windows or macOS.

Nothing is happening to -buildmode=archive. Or, for that matter,
-buildmode=c-archive. It's fine to link against as many
-buildmode=c-archive archives as you like.

Ian

[Peter W]

Thank you for your contributions to this language.

We will not see static reflection in golang2, right?

Anything else blocks the possibility of unloading implementation ?

Peter

[Ian Lance Taylor]

On Mon, Dec 27, 2021 at 12:58 PM Peter W <peter....@gmail.com> wrote:
>
> We will not see static reflection in golang2, right?

I'm not sure what you mean by static reflection, but I'm not aware of
any current plans to add any sort of reflection.

> Anything else blocks the possibility of unloading implementation ?

In Go it's trivial to start a background goroutine that doesn't stop.
If you unload a shared library that has a background goroutine, the
program will crash. I don't know of any reasonable way to work around
that.

Ian

> On Monday, December 27, 2021 at 2:17:42 AM UTC+2 Ian Lance Taylor wrote:
>>
>> On Sun, Dec 26, 2021 at 3:45 PM Peter W <peter....@gmail.com> wrote:
>> >
>> > I am trying to load, call and unload multiple go libs which are built with -buildmode c-shared in a c problem, it’s doable on windows, Linux and macOS ? Also I see -buildmode shared removal was rejected, what about -buildmode archive?
>>
>> You can't safely unload a Go DLL built with -buildmode=c-shared. That
>> is not supported at all. Sorry.
>>
>> Loading multiple Go DLLs built with -buildmode=c-shared ought to work
>> on ELF based systems like Linux. I don't know whether it will work on
>> Windows or macOS.
>>
>> Nothing is happening to -buildmode=archive. Or, for that matter,
>> -buildmode=c-archive. It's fine to link against as many
>> -buildmode=c-archive archives as you like.
>>
>> Ian
>

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/0f30e624-ea1a-4482-ab8b-d2566b9023a5n%40googlegroups.com.

[Peter W]

I seem to see some discussion about unloading lib that mentioned runtime reflect, probably it’s regarding shared not c-shared.

It’s possible to let the caller or callee take the responsibility of graceful shutdown?

Peter

[Ian Lance Taylor]

On Fri, Dec 31, 2021 at 3:04 AM Peter W <peter....@gmail.com> wrote:
>
> It’s possible to let the caller or callee take the responsibility of graceful shutdown?

I don't see how to do that in a reasonable manner. Any Go code can
start a goroutine, including code in some external package that you
without even knowing that it starts a goroutine.

If we provide such a facility, it will simply become an unending
source of bug reports that we won't be able to provide good support
for. Much like the plugin package.

Ian

[Sven Anderson]

On Mon, Dec 27, 2021 at 1:17 AM Ian Lance Taylor <ia...@golang.org> wrote:

Loading multiple Go DLLs built with -buildmode=c-shared ought to work
on ELF based systems like Linux.  I don't know whether it will work on
Windows or macOS.

 I have a related question on that topic:

How about support for a Go program that can load C modules (with Cgo and dlopen), and this module again is a Go c-shared library?

I would have assumed that does not work, due to conflicting runtimes, and it indeed crashes on my Mac. But after reading your statement here, I wonder if it maybe is supported on ELF based systems?

Thanks,

Sven

[Ian Lance Taylor]

I think that could work on ELF systems, but I haven't tried it.

Ian