Dynamic linking against individual golang standard libraries

[Parker Evans]

Hi,

Question about using dynamic libraries on Linux.  So I know that creating a shared .so for the golang standard library can be done via the following (or some appropriate cross compiling variant):

go install -buildmode=shared -linkshared std

When stripped of debugging information, this process results in a .so binary somewhere between 17MB and 26MB depending on version of golang and target architecture (amd64 vs. ARM for instance).  Not too bad for the ENTIRE golang standard library but still pretty big for a shared library especially on an embedded system.  One of the negatives for having such a large shared library on embedded Linux systems in addition to just the disk space is that you pay the penalty of the full shared library size in memory if at least one running application is using it (even if the application is only using a very small subset of the standard library packages).  So I have been investigating ways to generate and use shared libraries for each of the standard library packages (or appropriate subsets).  Is there a supported/easy way to do this?  Does anyone have any suggestions?

Going into the sync package for instance and building it using the shared build mode and -linkshared typically hangs in (cross-) compilation, whereas removing -linkshared will result in a shared library .so binary that is about 1.5MB in size.  To use it, I would probably need to specify appropriate linker flags to link against it (as -linkshared seems like it might be expecting libstd.so).  I am guessing that the build statically links in any go standard library dependencies that it uses (since I didn't specify -linkshared).

In order to do this kind of thing, do I need to manually figure out dependencies and then setup appropriate linker flags?  I realize this isn't too common a use case, but it could be very useful for controlling disk and memory usage in embedded systems using go.  Any help would be greatly appreciated.

Thanks,

Parker

[Ian Lance Taylor]

On Sun, Nov 20, 2016 at 1:22 PM, Parker Evans <pa...@cornell.edu> wrote:
>
> In order to do this kind of thing, do I need to manually figure out
> dependencies and then setup appropriate linker flags?

Yes, pretty much.

It may help a bit to look at go/build/deps_test.go.

Ian

[Parker Evans]

Alright, I will take a look.  Thanks for the response.

~Parker

[Parker Evans]

Alright, so I played around with this a bit.  It would appear that I can actually get pretty far by just compiling the full standard library as shared and then compiling the individual packages one at a time in the proper order.  What I am doing is captured in the program in this repository: https://github.com/ermergerd/go-dynlib

However, I hit a couple snags before I got all of the packages compiling (thus some are commented out in the application).  Specifically, I see the following linker panics when compiling log/syslog and crypto/ecdsa:

2016/12/02 22:05:56 Building package: log/syslog

2016/12/02 22:05:56 # /var/folders/5y/90kqcw0j72bgyj0kgxl1rnc40000gn/T/go-build689781855/liblog-syslog.so

panic: runtime error: index out of range

goroutine 1 [running]:

panic(0x19fa60, 0xc42000e130)

/Users/Parker/go1.7/src/runtime/panic.go:500 +0x1a1

cmd/link/internal/ld.(*deadcodepass).flood(0xc42003bcd0)

/Users/Parker/go1.7/src/cmd/link/internal/ld/deadcode.go:279 +0xe88

cmd/link/internal/ld.deadcode(0xc42048a000)

/Users/Parker/go1.7/src/cmd/link/internal/ld/deadcode.go:60 +0xe1

cmd/link/internal/ld.Ldmain()

/Users/Parker/go1.7/src/cmd/link/internal/ld/pobj.go:192 +0x1089

cmd/link/internal/arm.Main()

/Users/Parker/go1.7/src/cmd/link/internal/arm/obj.go:45 +0x19

main.main()

/Users/Parker/go1.7/src/cmd/link/main.go:30 +0x168

2016/12/02 22:06:05 Building package: crypto/ecdsa

2016/12/02 22:06:05 # /var/folders/5y/90kqcw0j72bgyj0kgxl1rnc40000gn/T/go-build010152613/libcrypto-ecdsa.so

panic: runtime error: index out of range

goroutine 1 [running]:

panic(0x19fa60, 0xc42000e130)

/Users/Parker/go1.7/src/runtime/panic.go:500 +0x1a1

cmd/link/internal/ld.(*deadcodepass).flood(0xc4204afcd0)

/Users/Parker/go1.7/src/cmd/link/internal/ld/deadcode.go:279 +0xe88

cmd/link/internal/ld.deadcode(0xc42049e000)

/Users/Parker/go1.7/src/cmd/link/internal/ld/deadcode.go:60 +0xe1

cmd/link/internal/ld.Ldmain()

/Users/Parker/go1.7/src/cmd/link/internal/ld/pobj.go:192 +0x1089

cmd/link/internal/arm.Main()

/Users/Parker/go1.7/src/cmd/link/internal/arm/obj.go:45 +0x19

main.main()

/Users/Parker/go1.7/src/cmd/link/main.go:30 +0x168

For reference, I am cross compiling the libraries to run on an ARM/Linux platform with the following command line:

CC=arm-linux-gnueabihf-gcc GOOS=linux GOARCH=arm GOARM=7 CGO_ENABLED=1 ./go-dynlibs -ldflags="-w -s -extld=arm-linux-gnueabihf-gcc"

Any thoughts on this method of doing things or the linker panics that I see for a couple of the packages?  I was a little confused why I needed to compile the full libstd.so before I could make any progress with building the individual packages as buildmode shared/linkshared.

Thanks,

Parker