Another call for shared libraries

[bolson]

I searched this forum for discussion about making a shared library or
doing dynamic linking in go, and didn't find much. Some discussion
about LGPL and a couple threads about difficulties linking against C
shared libraries.
I've heard Google internally static links _everything_ they do, but
the rest of the world still thinks dynamic libraries and debuggers are
a pretty good idea.
I think we should have Go shared libraries and dynamic linking for
technical reasons:
1. Disk space saved.
2. RAM space saved.
3. Plugin architecture.

1. Static linking takes up lots of space and makes binaries much
bigger than they need to be. Sure, disk is cheap, but it's not free,
and these things just seem embarrassingly large.
2. But all of that static linkage becomes 1000 times more expensive
when it's in RAM. If I'm running dozens of Go processes I could be
saving all the copies in memory of "fmt" and the other basic runtime
parts.
3. And almost as a side effect of dynamic runtime linkage you get the
ability to prorgammatically load libraries and use them as plugins.
Cool.

[DisposaBoy]

Not arguing against... but I don't think the mention of 'rest of the world' holds entirely true. I also, personally don't buy into points 1 and 2,

 at all.

[Archos]

http://harmful.cat-v.org/software/dynamic-linking/

[Johann Höchtl]

> the rest of the world still thinks dynamic libraries and debuggers are a pretty good idea.

How does static linking and debuggers relate? This is a real question. Or did you write this sentence with the knowledge in mind, that some of Go s core developers prefer print over GDB?

[Brad Fitzpatrick]

Brian,

I see 228 contributors in the CONTRIBUTORS file.

Each contributor has his or her own priorities about what they work on.

So far none of those 228 contributors have worked on dynamic linking support, suggesting that they have higher priorities.  I don't think anybody's violently _against_ shared libraries, though.  It's just nobody's top priority.

Maybe you'll work on it?

[bolson]

Brad, you make the reasonable and classic "do it yourself" open source
argument, so perhaps we should reinterpret my original statement as:
"Hey, I think shared libraries are neat because A, B, C. Does anyone
else similarly think they're neat? Is anyone working on shared
libraries?"
I _could_ work on implementing shared libraries for Go. It'd be more
interesting if there were good community interest in seeing this
happen. If the maintainers want it to happen they'll be more likely to
accept patches (and this would be a pretty big project with a pretty
big set of patches). If the community wants it to happen, the
maintainers might be more likely to want it to happen.
But, I got replies basically indicating that the vocal part of the Go
community _doesn't_ care about shared libraries, and one comment I
think implying that shared libraries may be a bad idea just not so bad
that it shouldn't be done anyway. The thing Archos linked was actually
kinda interesting. I'm curious about why shared libraries apparently
turned out to not save as much disk/ram as they might in theory.

[Ian Lance Taylor]

bolson <brian...@gmail.com> writes:

> I _could_ work on implementing shared libraries for Go. It'd be more
> interesting if there were good community interest in seeing this
> happen. If the maintainers want it to happen they'll be more likely to
> accept patches (and this would be a pretty big project with a pretty
> big set of patches). If the community wants it to happen, the
> maintainers might be more likely to want it to happen.
> But, I got replies basically indicating that the vocal part of the Go
> community _doesn't_ care about shared libraries, and one comment I
> think implying that shared libraries may be a bad idea just not so bad
> that it shouldn't be done anyway. The thing Archos linked was actually
> kinda interesting. I'm curious about why shared libraries apparently
> turned out to not save as much disk/ram as they might in theory.

I think it's desirable for 6g to support shared libraries. There are
some complexities. One of the advantages of Go is a simple
initialization mechanism (as compared to, e.g., C++). Initialization is
harder to understand when shared libraries are involved.

Note that gccgo already supports shared libraries, but it's not as clean
as it should be as I'm basically ignoring the complexities.

Ian

[Jan Mercl]

On Sunday, November 27, 2011 12:45:21 AM UTC+1, bolson wrote:

Let me try to turn the question (support shared libs or not?) into other questions/notes (condensed  a lot).

----

Case A: Shared libs from the C world (and compatible). Interacting with the C world currently uses CGO which produces the *necessary* bridges between Go and C. That's good because:

A1) The calling conventions do differ (and thread management/goroutines/split stacks/GC/...).

A2) CGO allows statically to consume C "interfaces" (defs) at Go's compile time.

Case B (hypothetical): Shared libs produced by some future Go compiler.

----

Estimations.

Dynamic loading of case A libs:

- Generating ABI "bridges" at runtime - feasible, though probably suboptimal compared to the CGO way.

- Good run time performance of "type bindings" generated at compile time - unfeasible or unrealiable (DLL hell).

Dynamic loading of case B libs:

- Not feasible w/o breaking static type checking and/or reflection (and I suspect even more trouble ahead).

----

Conclusion: I expect much more problems, some perhaps not solvable in any good way, than any possible gains from dynamic linking.

[Kyle Lemons]

> Brad, you make the reasonable and classic "do it yourself" open source

Hmm, Go is an open source project...

> argument, so perhaps we should reinterpret my original statement as:
> "Hey, I think shared libraries are neat because A, B, C. Does anyone
> else similarly think they're neat?

Yeah, lots of us would like some sort of dynamic linking.  I don't personally want shared libraries for the stdlib, but I would be interested in dynamically loaded modules for plugins and whatnot.

> Is anyone working on shared
> libraries?"

No.  As was said.

> I _could_ work on implementing shared libraries for Go. It'd be more
> interesting if there were good community interest in seeing this

It's only interesting if someone is committed to working on it.  To me, at least.

> happen. If the maintainers want it to happen they'll be more likely to
> accept patches (and this would be a pretty big project with a pretty
> big set of patches). If the community wants it to happen, the
> maintainers might be more likely to want it to happen.

If you follow the contribution guidelines and hash out the design first on golang-dev, you will know pretty well what the committers think about the design before you write any code.

> But, I got replies basically indicating that the vocal part of the Go
> community _doesn't_ care about shared libraries, and one comment I

Because you can largely get along without them.  And unless someone is committed to making it happen (or has a unique suggestion), lots of people will think "here we go again" and move along.  That's true of generics and dlso and a half-dozen other common requests.

> think implying that shared libraries may be a bad idea just not so bad
> that it shouldn't be done anyway. The thing Archos linked was actually
> kinda interesting. I'm curious about why shared libraries apparently
> turned out to not save as much disk/ram as they might in theory.

Search Google for "dll hell" or "problems with dynamic linking" or something; there are lots of articles floating around.