Recommended way to distribute pre-compiled C-dependencies for modules using CGO ?

[Jan]

hi,

I'm developing a couple of ML framework/libraries for Go that depend on C/C++ code. Once C-libraries dependencies are installed, the CGO integration work great.

Now, for end-users that just want to use these Go libraries, having to figure out how to manually build and install those C/C++/Rust dependencies is a hassle -- sadly each one with a different type of build system.

I offer pre-built `.tgz` files (for a limited set of architectures) with the required `.h` and `.a/.so` files along the releases, which facilitates. But it's still a hassle to install -- and no auto-uninstall if someone is no longer using the Go module.

I was wondering if others have figured out how to handle this in a nicer way.  Is there a recommended way to distribute prebuilt CGO dependencies ?

I like how Python wheels (`.whl` files) solve the issue, by including the pre-built libraries in a sub-directory of the python library installation. I was hoping there would be a similar way (maybe with a separate tool) to integrate with `go.mod`. 

Any pointers, thoughts or suggestions are very appreciated.

Thanks!

Jan

ps: While searching I saw similar questions, but none that exactly answered this aspect of distribution. Just in case, apologies if it's a duplicate question.

[Peter Galbavy]

I use a docker container (golang:bullseye seems the most compatible) and build static binaries. This is feasible as long as your dependencies do not require dynamic loading of dependencies.

Poor (my Dockerfile skills) example here: https://github.com/ITRS-Group/cordial/blob/main/Dockerfile

Peter

[Jan]

Thanks Peter, but I don't follow ... I also use docker to build the pre-built binaries.

But how do you distribute them, so folks using your library don't think to install them ?

[Jan]

Edit: ...so folks using your library don't need to install them manually ? What about uninstalling ?

[Peter Galbavy]

If you link statically, e.g. from my Dockerfile:

  go build -tags netgo,osusergo --ldflags '-s -w -linkmode external -extldflags=-static'

then the libraries are in the single binary. I also then use "upx" to compress the resulting binaries to save a little space. The build tags are there to not require GLIBC for user and dns lookups.

[Peter Galbavy]

Hang on - I missed your original statement that you are building *libraries* yourself. I am building executables. Sorry for the confusion and my too-fast scan reading!

[Jan]

Yep, I want a solution on how to distribute C-dependencies of libraries that use CGO, so end users don't have to manually sort out dependencies of various libraries.

Thanks for trying though.

[Richard Wilkes]

It isn't a great solution, but I currently include the built library files and necessary headers in the git repo alongside the Go code. You can see an example here https://github.com/richardwilkes/unison/tree/main/internal/skia where I include the skia library I built for use in my UI framework, unison.

The main downside of this is bloating the git repo with the binary .a and .dll files... but I've not found a better way to handle it. glfw, which unison also depends upon, does something very similar.

- Rich

[Tim Casey]

I would put down Makefile includes for the supported targets and build each target at a time, as cross compiled shared libraries.  This is easier for linux, harder for windows.

But, you can build/export from docker containers as part of the overall build process to allow simulated build environments.   At some point, you may come to a dependency in which you have to build the included C code for it to run in platform dependent libraries.  At this point, you probably will want to build on the environment as you go.  You can cross target *.o, but build the *.so on the specific target.

You could have separate repos and distribute the C code from a dependency point of view.  This would be a tgz of 'shared/$ARCH/$COMPILED_LIB", or some such.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/3b03b9e1-655e-452a-9b97-e9471b9b9dc1n%40googlegroups.com.

[Jan]

Thanks Richard. Indeed, as you pointed out the downside is the bloating of the git repo, but it makes sense.

But does the user have to manually clone the repository and move the `.a` file to, let's say, /usr/local/lib, or does a simple `go get` magically does everything ?

[Jan]

Thanks Tim, but my goal is to make things transparent and easy to the users of my libraries. So I don't want to force them to run Makefiles (in my case large Bazel builds for one project, Makefile for the other), install C++/Rust build tools, deal with dockers, etc. Ideally, `go get` would just take care of everything. If that is not possible, I'm wondering what is the easiest next thing.

The reasoning is that a largish project often depends on dozens of libraries, if we need to do this kind of stuff for every library, its too much friction. Go tools are awesome ('go get' works so well for me) when there are no CGO dependencies ... alas, not everything is implemented in Go.

Indeed, pre-built binaries is platform dependent -- and requires separate files/tgz per architecture, as you point out. How to have these dependencies installed, and picked up by go compiler automatically (setting up of CGO's CFLAG, LDFLAG LD_LIBRARY_PATH, etc), and be removed if no longer used ? With minimum user headache/work...

cheers

Jan

[Tamás Gulácsi]

Can't you build (make go build for you) those libraries?

For example, see github.com/godror/godror just includes the sources of that third party library in an "odpi" subdir, and with

```

/*
#cgo CFLAGS: -I./odpi/include -I./odpi/src -I./odpi/embed

#include "dpi.c"

*/
import "C"

```

it is compiled automatically.

Caveat: for "go get" to download a directory, it must include a sth.go file ("require.go" in the odpi/* subdirs).

But it may work that your precompiled libs in a subdir, with a mock .go file gets downloaded.

But how will it found by your lib/app ?

Tamás

[Jan]

Thanks Tamás, I may not be understanding correctly, but after taking a look at github.com/godror/godror, and the odpi subdirectory,

I see it is including all the `.c` files on the fly.

A couple of reasons immediately come to mind, that make this not a generically valid option:

* ODPI library is all C code (see src) so it works including in Go: my dependencies are C++/Rust code, for which I write a small C wrapper (or for Rust just `extern "C"`). Also architecture dependent compilation is different in C++/C/Rust code ...

* The C++ libraries I'm including have sub-dependencies of themselves (one of which is llvm). It uses Bazel to organize it, and to manually move all the required C++ files to a directory would be years of work :) Plus would require me to slave to maintain things in sync. 

* The C++ libraries take hours to compile ... I don't want to impose this to users of my libraries.

I think the only way to work this out is distributing the pre-compiled C++/Rust libraries, so the Go simply refer to them (and we get the fast compilation times from Go). But then, how to best distribute them in an automatic fashion, so that users don't need to one by one figure out how to install them (and clean up them later if they are no longer using...) ?

Or maybe there is another convenient way I'm not seeing ?

[Tamás Gulácsi]

Neither I see a convenient way.

BUT if you add a .go file into the directories where your precompiled libraries live,

then "go get" will download them too (and only those dirs that have .go files in it).

So your next mission is to prepare the right #cgo CFLAGS LDFLAGS incantations to use those libraries.

[Jan]

Thanks Tamas, this is useful information. 

One of my libraries is using a `.a` library -- as opposed to `.so`, which is another level of complexity, since the library has to be available in runtime, not only in compile time -- and I'm going to follow your "incantation" suggestion.

[Bruno Albuquerque]

I guess you switched things here. Shared libraries (.so) need to be available at engine. Static libraries (.a) are bagged into the binary.

-Bruno

--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/a00b3a7b-200d-44ce-8060-913d9ac3c6cen%40googlegroups.com.

[Mike Schinkel]

Hi Jan,

I'm going to go out on a limb here and suggest looking at using Go's `embed` feature?

https://blog.jetbrains.com/go/2021/06/09/how-to-use-go-embed-in-go-1-16/

I have not tried it with C libraries so there may be numerous reasons why it may not work for your needs. For example, I do not know what is required to "install" the libraries so that might be your sticking point that embed would not address. But if building on the user's machine is the real problem and you can distribute pre-build binaries then this might work for you. 

Basically Go reads a `//go:embed` comment, converts your files into Go source code, compiles that code, and then provides a package that allows you to write those files to disk from an your Go app before you need to load the libraries.

Maybe this will work for you?  If yes, would love to hear back how it worked out.

-Mike

[Marc-Antoine Ruel]

I second Richard's suggestion. I used the same trick for https://github.com/periph/d2xx. This git repository contains a minimalist shim for the .a libraries and nothing else. It's designed to compile on any platform. It falls back to a scaffolding if the corresponding .a library is not present for the OS-arch combination or if CGO is disabled. It makes testing easier.

Then a proper package under https://github.com/periph/host/tree/main/ftdi exposes a higher level abstraction for the user.

M-A

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/ca2d8979-f741-4b81-9099-8672bc5e2312n%40googlegroups.com.

[Justin Israel]

On Tuesday, October 17, 2023 at 10:40:33 AM UTC+13 Marc-Antoine Ruel wrote:

I second Richard's suggestion. I used the same trick for https://github.com/periph/d2xx. This git repository contains a minimalist shim for the .a libraries and nothing else. It's designed to compile on any platform. It falls back to a scaffolding if the corresponding .a library is not present for the OS-arch combination or if CGO is disabled. It makes testing easier.

Then a proper package under https://github.com/periph/host/tree/main/ftdi exposes a higher level abstraction for the user.

With only headers and static libs in the thirdparty directory, is this package "go-gettable"? 

Will go make the subdirectory available in that case? It usually ignores if there is no Go source files. 

[Marc-Antoine Ruel]

Sorry I wasn't clear. The static libraries are in a subdirectory because the user should not care and these libraries are effectively third party code.

This declares the generic code available everywhere:

https://github.com/periph/d2xx/blob/main/d2xx.go

One set of files declare the import that is OS and CPU arch appropriate, e.g. 

https://github.com/periph/d2xx/blob/main/d2xx_linux_amd64.go

Then another set of two files defines the OS specific code. In practice, POSIX and windows;

https://github.com/periph/d2xx/blob/main/d2xx_posix.go

https://github.com/periph/d2xx/blob/main/d2xx_windows.go

A CGO_ENABLED=0 build gets instead redirected to

https://github.com/periph/d2xx/blob/main/d2xx_posix_no_cgo.go

but not on windows because this package uses dynamic linking on windows. Adapt as appropriate in your use case.

I hope this helps.

M-A

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/146f101d-8ab4-40ec-a03d-ea23aa84bf74n%40googlegroups.com.

[Justin Israel]

On Wednesday, October 18, 2023 at 3:12:29 AM UTC+13 Marc-Antoine Ruel wrote:

Sorry I wasn't clear. The static libraries are in a subdirectory because the user should not care and these libraries are effectively third party code.

Actually, you were totally clear. Sorry if my comment didn't make sense. I get that the user doesn't need to worry about the third_party directory with the header and pre-built static libraries. And I saw how you arrange to pick up the right ones in the code. My point was that from previous experience, "go get" would prune away directories that do not contain go source code, after it performs the clone from the remote, into the module cache. So I would have expected that when someone uses your library as a dependency, it would have omitted the needed third_party directory, for not having any Go source files in it. But when I tested it just now, it does seem to retain the subdirectory and link correctly. Maybe this was improved as of recent Go releases. 

In earlier versions of the Go tool, I would have either included a dummy Go source file in the subdirectory, or just kept the non-go files in the root with the rest of the Go source.

Maybe someone else can clarify if this workflow has been improved?

[Jan]

Thanks for all the replies. 

Unfortunately, as I found out today, including the `.a` (static libraries) in the Github just got to its limit (100Mb), so I don't think this is an option anymore :( I'm going to investigate the Gihtub LFS (Large File Storage), not sure how well it will integrate with `go get` though -- I've never used it before.

 Interesting to know that an empty Go package file is no longer needed to fetch a directory with binary dependencies (handy also for go:embed in cases where the binary files are on a separate directory).

Still for my Go library with a `.so` dependency I'm not sure yet what is the answer. I'm thinking about creating another tool that goes over all the dependencies of module, and automatically downloads release files to a configured directory ... so at least all one needs to do is execute one command and get all dependencies of all libraries installed. Any thoughts ?

@Mike Schinkel: the go:embed is used in a different context (when embedding blobs of binary). We need prebuilt binary libraries that will be 

linked with the Go code during compilation. Preferably something that will transparently work with `go get` / `go build` / `go install`.

cheers

[Jason E. Aten]

> including the `.a` (static libraries) in the Github just got to its limit (100Mb)

I have used bzip2 to compress libraries that are too big for Github. If that gets them under the limit, then great. 

Just provide installation instructions or a Makefile target that decompresses them once checked out.

(Or xz? Apparently it can compress more than bzip2; but I have no experience with it. https://en.wikipedia.org/wiki/XZ_Utils )

[Jan]

Thanks @Jason, but the point was exactly not need to do anything extra: no manual unzipping of a file, or manual Makefile/Magefile.

Let's say project in Go links some 30 external modules, 5 of them have their own specific steps that need to be read, understood and and run to install them ... very quickly the user just gives up from installing/compiling the thing, with good reason ... 

The Go toolset solves that beautifully for Go only projects. But at least in my line of work, that's not feasible, I need to interface with libraries that in turn require other libraries in C++ (OpenXLA/llvm for JIT), Rust (HuggingFace Tokenizer), C (Gtk), etc.

cheers

[Jason E. Aten]

What I would do for that case would be to deliver the users a either a VM image (heavy), or a docker container (lighter) with batteries included inside. 

There is often little need to write a docker file unless you really want. Just get everything working inside docker and "docker commit" it to an image. I do this often for projects that need python/R + 100s of libraries.  Its really the only sane way to deliver sprawling dependencies that assume they can write all over the filesystem.  

One hint: I recently did this for code that wanted to output graphics, and that needed X11 (or Xvfb), and that wanted systemd, which is available in docker but used to be discouraged so is not available in many base images. So I would recommend starting an docker image that already supports systemd and X11, if graphics (or x11vnc, say) is ever going to be desired. Its a pain to add after the fact. Podman claims to be useful for this, but I found in immature and not really production ready when trying to run it on blah standard Ubuntu 22.

[Jan]

That works if what I'm doing is a end product (a server or something). But what I'm doing is in itself a library. Imagine if every library offers a docker/VM image, how is the end-user supposed to merge those docker images ? The docker is also not a viable/ergonomic solution in my case. I mean ... actually I do offer a docker for one of my projects, along with Jupyter, a Tutorial and demo. But generically, it is a library, and I would like it to be easily `go get`able. Or at most with one simple/standard step.

Yes, I hear you with respect to unnecessary extra dependencies (X11) -- at least make them optional, right ? But In my case the dependencies are essential, and the docker doesn't really solve the problem...  But also I'm running out of hope that there would be anything to solve it, I'm almost thinking I should write something myself.

[Jason E. Aten]

It is still rough around lots of edges, but you are wanting to write your own deployment system, you might find in https://nixos.org/ some inspiration.

It generalizes the idea of cryptographic hashed based dependency management to, well, everything.

Sadly not that easy to use; it may be harsh to inflict on end users.  But it tackles the reproducibility issue (it works on my machine! but not yours?) head on.

You might profitably write some glue between Go and Nix that provides a nice end user experience.

[Robert Engels]

Better to rewrite the native portions in Go. 

If you’re not going to do that, just give instructions on how to build and let the app builder figure out how to package it. 

On Nov 3, 2023, at 1:47 PM, Jason E. Aten <j.e....@gmail.com> wrote:



--

You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/149f28a5-417e-4642-9c04-13f0254e2dcbn%40googlegroups.com.

[Luke Crook]

For Windows, could you package and distribute the prebuilt binaries using an MSI file? (https://nsis.sourceforge.io/Main_Page)

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/5E033742-EDBD-4D44-BD09-60927C27448E%40ix.netcom.com.

[Konstantin Ivaschenko]

@Jan You haven't thought of a good way to do it?

суббота, 4 ноября 2023 г. в 05:23:51 UTC+3, Luke Crook:

[Nagaev Boris]

On Fri, Oct 20, 2023 at 4:59 AM Jan <pfe...@gmail.com> wrote:
>
> Thanks for all the replies.
>
> Unfortunately, as I found out today, including the `.a` (static libraries) in the Github just got to its limit (100Mb), so I don't think this is an option anymore :( I'm going to investigate the Gihtub LFS (Large File Storage), not sure how well it will integrate with `go get` though -- I've never used it before.

Hi Jan!

`.a` static library is an archive of `.o` files. It can be unpacked
and repacked into several `.a` files so that each of them is less than
100Mb GitHub limit.

ar -x libbig.a
ar -r libsmall1.a obj1.o obj2.o obj3.o
ar -r libsmall2.a obj4.o obj5.o obj6.o

See https://stackoverflow.com/a/53522978

Then link all `.a` files from #cgo.
You may need to link the libraries twice:

#cgo LDFLAGS: -L${SRCDIR} -lsmall1 -lsmall2 -lsmall1 -lsmall2

Alternatively you can figure out dependencies between .o files, run
topological sort before splitting and make sure that small2 does not
depend on small1. Then no need to link them twice.

See https://stackoverflow.com/a/409470

Also `.o` files can be stripped using the `strip` tool unless you need
debugging. This can result in dramatic size decrease.

> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/9e56ac27-386d-41d0-a520-263e3402c6d2n%40googlegroups.com.



--
Best regards,
Boris Nagaev